I also discovered that using a foreign key of an object returned by a Django QuerySet as a dictionary key prompts Django to look the actual data up. I had something like the following:

class WeekData(models.Model):
    artist = models.ForeignKey(Artist)
    plays = models.PositiveIntegerField()
    ..

tracking = defaultdict(int)
for week in WeekData.objects.all():
    tracking[week.artist] += week.plays


The dictionary update was taking ages. Confused, I enabled Mysql’s logging and discovered 25,000 lines of the following..

SELECT `id`, `name` FROM `muncher_artist` WHERE `id` = 22
SELECT `id`, `name` FROM `muncher_artist` WHERE `id` = 23
SELECT `id`, `name` FROM `muncher_artist` WHERE `id` = 24
SELECT `id`, `name` FROM `muncher_artist` WHERE `id` = 25

At which point I realised that using week.artist as the key here looks up the artist every time, meaning 25,000 useless database queries and a really, really slow function. Perhaps I was being too hopeful in my expectation that Django would be clever.

Changing the last line to:

tracking[week.artist_id] += week.plays

sped the function up by a factor of ten and lets me produce images like this in reasonable time:

It’s showing which artists occupy which chart positions as the weeks go by. The dark black line is for Fleet Foxes .. seems I went pretty mad for them :o

Spent some time moving the last.fm charts thing to Python, because Python is tasty, PHP is clumsy, and matplotlib is a superb piece of kit.

Here’s a histogram of the number of tracks I’ve listened to each week for the last few years:

(x is the number of tracks in a week.) I looked at a few friends’ and there seemed quite a split between those roughly following a Gaussian distribution and those following something rather more exponential. I should experiment with different features of matplotlib, maybe narrow those bins a little. I’ve lots of little plans for other graphs and combinations.

Finally, there’s a great little piece about a quirk of the number 1/89 here. Sum the numbers of the Fibonacci Sequence in the following manner:

  .01
  .001
  .0002
  .00003
  .000005
  .0000008
  .00000013
  .000000021
  .0000000034
  .00000000055
  .000000000089
  .0000000000144
        .
    +   .
        .
----------------
  .01123595505...    = 1/89

The link has an outline proof about why it’s true.

Same process as before – when you first create each kind of chart it’ll take a while to fetch your data, subsequent charts will be created a whole lot faster. Eventually I’ll make a script to purge the data stored for anybody who hasn’t used the program in a couple of months so keep popping back if you get fed up with the waits.

As always comments are welcome. Oh and suggestions for a better name too..

I’ve made it automatically updating, so you only need to add your data once the very first time you visit, and after that it’ll check whether there should be new charts available. Maybe it should only happen a few days into the week – LastFM never get the charts out on time after all..

That and I added a few extra quick links, to 3/6/12/24 months previous. No more ‘wait a month for charts to refresh’ ;-). I should totally see if I can make this into a greasemonkey script..

PHP’s strtotime function is no end of brilliance – strtotime("-6 months") and strtotime("last sunday") are genius conceptions.