Andy Balaam’s Blog

I’ve made a new release of FreeGuide today, which contains minor bug fixes.

I changed the stability rating on sourceforge to “Production/Stable” (previously it was “Alpha”). Maybe I should bump the version number next time to 1.0. Certainly all this 0.10.x stuff is crazy.

I managed to get rid of quite a bit of unused and unworking code this release, so I’m feeling somewhat more positive about developing FreeGuide. My next plan is to spruce up the ant build so you can easily use it in Eclipse, and convert the whole lot to a single JAR, which may improve startup time.

The program seems to work for quite a few people, and I don’t change it much these days, so it might be time to call it 1.0. The only major feature I am planning (one day…) is to combine the Find dialog with the vertical view and possibly the favourites editor to make them different views onto the same underlying implementation. I’d like to make all of them show filter results as you type.

Other than that all I really want to do it fix user pain in terms of bugs and UI wrinkles.

Of course, I am bound to get a bug report immediately I have posted this telling me my fix breaks everything, but for the moment I am chuffed that I found, tested, and fixed a genuinely difficult bug.

I am particularly proud because I wrote an automated test to ensure it can never happen again, and I used that test to make the debugging process much easier than it otherwise would be. The code that reads, processes and stores listings in FreeGuide is a spider’s web of interfaces and helper classes (because of the arguably over-engineered plugins framework used for all the moving parts), and tracking this down with plain old-fashioned debugging would have been a huge job.

Anyway, I bet you are dying to hear what the bug was, aren’t you?

When you have a programme already stored in FreeGuide, and then a new one comes along that overlaps it, the old programme is deleted. For example if we start off with:

... 19:00 Top Gear ................... 20:00 Charlie and Lola .............

but then later download listings again and get these programmes:

... 19:00 Pocoyo .. 19:15 Round the World ...... 

Then Top Gear will disappear, and be replaced by the 2 new programmes. In fact, any old programme that overlaps any new incoming programme will be automatically deleted.

At least, that is what is supposed to happen. In fact, the real situation is a little more complex because the programmes are stored in separate files (.ser files) for different days and times. Actually, there are 4 files for each day, named things like “day-2008-09-15-A.ser”, where the suffix A, B, C and D indicate which part of each day a file is for.

So imagine what happens when the first set of programmes comes in looking like this:

19:00 Programme 1A ......... 21:15 Programme 1B .. 21:30 Programme 1C ........... 22:00

and then the second comes in like this:

19:00 Programme 2A......................................... 21:45 Programme 2C .. 22:00

So obviously the old 3 programmes should be completey deleted, and the new 2 should be what you see.

But you don’t. In fact what you see is programme 1B and programme 2C, before 1B and between the two. Weird huh?

“Why?” I hear you ask. Well, it’s simple when you consider how the programmes are split into files.

Programme 1A goes into file day-2008-09-14-D.ser, and programmes 1B and 1C go into day-2008-09-15-A.ser.

[Side note: this is true in this case because the bug reporter is in the GMT -0400 timezone and the file boundaries are quarters of a day in GMT.]

Then, when the new programmes come along, 2A goes into 14-D – wiping out 1A, and 2C goes into 15-A – wiping out 1C but not 1B.

Then, when the files get read back in again later, 2A is read from 14-D, but then 1B is read from 15-A, wiping out 2A, and finally 2C is read in as well from 15-A, so we end up with 1B and 2C.

How to fix it? Well, what I did was leave everything as it is, and then do the final read in the reverse order. This means we read in 1B and 2C, but then we read 2A later, and it wipes out 1B, leaving 2A and 2C as we would expect.

Neat fix eh? It works because this kind of wrongness in the .ser files will only exist when a programme hanging off the end should have wiped out something in a later file. Because programmes are classified into files by their start time, they can only hang off the end of a file, not the beginning, so reading the files in backwards will always read the hanging-over file last, wiping out anything which should have been wiped out earlier.

There is a little bug/feature remaining, but it only applies when you get some really weird listings from your provider. If you had a programme like 1A (19:00 – 21:15), and downloaded new listings, which ONLY contained a programme overlapping it, but falling into a later file (so maybe it starts at 21:00), and didn’t contain any programme starting at 19:00, then the backwards reading would mean you would never see your new programme because it would be wiped out by 1A.

This is a very unusual case though, since normally if you get a new programme at 21:00, you will also get new programmes leading up to it, if only to reflect the fact that 1A is now a different length. So this is really a theoretical bug, which explains why I’ve decided not to fix it…

Anyway, by the time I’d fiddled with my test for this to get the bug to trigger (which took a long time – working out which bits to fake out and which to test at all was tricky), the actual fix was easily implemented (1 line of code I chose to break out into 3), and then validated in a single click.

Just in case I hadn’t mentioned it, I love tests.

So let’s quietly forget FreeGuide 0.10.8, which basically didn’t work.

I fixed the problem (which was some nasty threading thing) by making the initial “Choose Channels for XMLTV” call from the first time wizard not do the clever stuff of displaying errors to the user. Instead it dumps any errors to the console (which is an improvement over swallowing them completely as it did before), and when you run “Choose Channels for XMLTV” from the menu, or “Configure” from the Options screen, you still get the errors displayed in a nice dialog as in 0.10.8.

More excitingly, new in this release: some unit tests!

You can run them by doing this in the src directory:

java freeguide.test.slow.FreeGuideSlowTest

They test the fixes I made in 0.10.9 for XMLTV files that have certain attributes missing, and have different date formats.

The date formats were supposed to be already working, but a fix I made several years (yes, years) ago broke them. Why didn’t I catch the problem? No tests.

The second age of FreeGuide is coming. I shall call it The Time of Testing.

I am still working slowly on moving FreeGuide forward. Somehow it seems my itches for FreeGuide are all about making it less annoying for people who are trying it the first time. I guess this is motivated by my desire for world domination.

Anyway, we are one small step closer to my mum being able to use FreeGuide – when the “Choose channels” step (i.e. the XMLTV grabber configuration) goes wrong, you can now see a real genuine error message, and hopefully figure out what went wrong.

Actually, it always used to work that way but the error-catching got refactored away at some point. Anyway, I am slowly taking the ground back…

As I do more and more test-driven development at work I am becoming completely addicted. For this FreeGuide code I wrote a couple of unit tests but they are not within a proper framework, and can’t be launched easily as a test suite. I am considering JUnit.

I also want to set up some component-level tests e.g. for downloading listings for each country and checking everything works as expected. It’s brilliant fun having tests in place, but when you have as little time as I have for FreeGuide at the moment, it’s difficult to decide to spend a long time working on a test framework when I could be fixing a “real user problem” or adding a cool new feature.

But I’ve got the testing bug badly, so watch this space.

On the subject of distributed source code management, Dan Watkins has just informed me that the launchpad team have created a bazaar branch of FreeGuide’s code, so if you’re into that kind of thing, you can download the code from that instead of our central subversion repo.

The link is here: http://code.edge.launchpad.net/~vcs-imports/freeguide-tv/trunk.

If you’re into git, I’d suggest git-svn. It’s what I use for FreeGuide development now. Let me know if you have trouble getting it working.

Check out a release candidate here: http://freeguide-tv.sourceforge.net/rc/ .

So far I’ve only uploaded an RPM, but hopefully there’ll be a Windows exe and some other packages soon.

Update: Windows installer now uploaded too.

Test it! It’s got better capturing and reporting of download errors, more sensible “download in background” and “show output” behaviour, and its list of XMLTV grabbers is synched with the latest XMLTV (0.5.50).

However, apparently XMLTV 0.5.50 is causing problems in FreeGuide for at least some people, so it may not be a flawless experience…

I really must make a one-button build for Ubuntu .debs, like I have for RPMs. I have been using Ubuntu for some time now…

I’d forgotten to update the version page on the FreeGuide web site for 0.10.5, so users weren’t notified of that release. I’ll do it for 0.10.6, and hopefully get some stats together to find out how many people are using it. Now that the stats are working, we may be starting to get some better information.

Yes, I couldn’t sleep tonight.

No, I won’t be very clever tomorrow.

But hey, I was clever tonight – dipping into some of the spaghetti that makes FreeGuide tick these days and not only making a change that works, but also making things a tiny bit better – _that’s_ an achievement.

After over a year, and several U-turns, the next version of FreeGuide is out.

I was dead-set on getting recording functionality into it, and when some students turned up asking to help, I directed them towards doing that. They produced a reasonable framework for how to do it (without any of the dirty details of actually recording stuff), but we ended up throwing it away in favour of the way Alex wanted to do it, which was as an extension of the current reminder code.

This way of doing things never reached a stable state, so the trunk was festering for a long time with broken code in it that lost your settings, and tried to record programmes but started too late, and with loads of other usability bugs that I’d hesitate to call minor since any one of them could put off someone trying the program for the first time and persuade them never to come back.

In the end, since this just wasn’t getting fixed, I decided we needed to give up on recording and get back to what we’re good at, which is being a TV guide. There is, of course, quite a lot of overlap in these two functions, but I just didn’t have time to think through properly what a recording program should do, and how to move FreeGuide over to a sensible model.

So, I reverted the recording code, and it’s there in the SVN history if anyone wants to resurrect it.

Meanwhile, the thing that motivated me to do this and actually get a release out was the fact that the US listings provider Zap2It has wound up their service, and it’s been replaced with a new one, called Schedules Direct, for which you have to pay money.

Hopefully, any US and Canadian users of FreeGuide should be able to download the latest version of FreeGuide (and update to the latest XMLTV if they’re on Linux) and start viewing their listings as soon as they’ve paid for them from Schedules DIrect. Do let us know on the mailing list if this isn’t the case.

So, hopefully the crisis has been averted, and now we can get back to doing things right. I’ve implemented the Benevolent Fascist Dictator rule, and so far, trunk is better than 0.10.5, and we could make a release from it any time. I plan to make one every couple of months, to keep things ticking along.

FreeGuide 0.10.5 contains millions of small features and bug fixes, that I didn’t manage to keep up with while trunk was broken. The next release will be after a much shorter delay, and will have a much better change log.

Why not try the latest SVN version of FreeGuide, which contains lots of bug fixes over 0.10.4? There are detailed instructions on how to Build FreeGuide from SVN, and do ask on the developers’ list if you have problems.

I’ve fixed the horrible bug that deleted all your preferences, and I think the code’s in pretty good shape, with all the recording code gone, but lots of nice additions put in over the last year.

Try it out, and let us know on the developers’ list how you get on – good or bad.

FreeGuide has been stalled for about a year because the basic code for recording programmes has been sitting in SVN, but it contained far too many bugs to be useable, even by the developers. This kind of situation completely takes the wind out of development, since any little bug fixes or features people want to send it are either against very old code, or never get written because when they download the latest code it doesn’t work.

I am more and more of the opinion that the code in your trunk (or HEAD, in CVS terminology) should ALWAYS by _better_ than the latest release. I.e. it should only contain bug fixes and properly tested new features. Anything else (i.e. code that is still being knocked into shape) should live in a branch. Branches are pain, but not as much pain as your project dying because new developers who try and do the right thing (download the latest code) are confronted by non-working code and move on to something more fun.

So, I’ve reverted the recording code. If we want to resurrect it, we can branch from just before where I reverted, and then merge from the trunk into that branch any other fixes that apply.

The next release of FreeGuide will be from the trunk, and will contain bug fixes and small new features submitted by a couple of people who have stepped up to do the work. I hope, when my life settles slightly, that I might even be one of them.

New baby resolution: never allow non-completely-working code in trunk. Without new developers, your project will die.

Small hypocritical extra note: I completely broke the code in trunk by reverting the recording code, so now it loses all your settings when you exit. Use with extreme caution. This, of course, completely breaks the above resolution. I will fix it when I can, and in the meantime, Rick is looking into it, so hopefully it will be fixed soon.

A long time ago I added functionality to FreeGuide so people could opt to report their use of FreeGuide to under a username, or anonymously. This all happens as part of the latest version check, and you can turn it off, obviously.

Quite a long time ago, Alex broke this functionality, and I haven’t yet made a release that fixes it. Stupid, really, because the usage statistics we’ve got are all-but useless:

Hopefully I’ll get a release out soon that fixes it, and these graphs will become more sensible.