About the Last.fm Spring Graph Sketch

Description, sources, and notes on the Last.fm spring graph sketch.

What Last.fm Graph Does

First, it gets a list of the top 50 artists for a Last.fm user. These are shown as the gold circles. The size of the circles are based on the total number of plays so a users favorite artist is probably the biggest circle. (The first time it runs for a new user, it is fetching lots of pages from Last.fm and you will probably notice pauses. It will fetch this data locally on the second run through an will be much smoother. If it never gets smooth at all, your computer might just be slow, or you might benefit from the latest version of Java (this was developed using version 1.6 update 7).)

Top50

Then it gets a list of similar artists for each of the top 50 artists. The similar artists are the blue circles. They are connected to the artists they are similar to by 'springs' that vary in length and strength according to how strong the similarity is. A blue circle has to connect to more than one other circle to become visible.

similar

Then it gets a list of weeks for which Last.fm has data. The weeks are represented as lighter blue lines on a dark blue line. Gaps are usually when a user is on vacation, or just doesn't play any music on their computer.

timeline

Then, for each week where there's data, it gets a list of the artists played. The labels for these artists are lit up from week to week. These weekly artists are shown in orange if they aren't already in the top 50. A list of similar artists is retrieved for any new artists and these new blue circles are added as well. The size of the circle is based on number of plays and will grow from week to week if the number of plays increases.

weekly

The final result will be what's referred to as a network spring graph. The artists will probably be arranged in clusters of styles. In my drawing, I can clearly identify different areas of jazz, rock, pop, Japanese pop, and French contemporary music. It's interesting to see how the different styles interrelate and what artists are in the areas between styles. It's also interesting to see what I was listening to when. Some weeks are heavier on the jazz than others and sometimes I'm listening only to stuff tightly clustered around my favorite bands.

You can enter a new user id by clicking the square, or add a user to the current drawing by clicking the blank square. It's fun to see where your tastes overlap with your neighbors. A group time line shows what weeks Last.fm has data for.

It can take a while to get all the data especially for more than a couple of users at once. The data is cached to disk, though so things should be quicker when the time line repeats. Before you give up, go walk around the block or call a friend to say hi. You might have something groovy waiting when you get back.

It is possible to get 'floaters' of two or more nodes that aren't connected to the main graph. They will slowly drift away and can make the scale go wrong. You can remove any node by right clicking on it. Single floaters are removed from the screen every so often automatically. A node can come back if it becomes connected to something by new data.

How it's Made

The first version of this sketch was done entirely within the Processing environment (version 0143). I have found Processing to be a wonderful way to quickly get the computer to draw something and I can't recommend it highly enough for anyone interested in making the computer do something visual. That said, I had lots of trouble with the readlines(url) method crashing every few dozen requests to Last.fm. Processing is based on Java and with a fairly small investment of work, I was able to integrate the Jakarta Commons HTTP Client. This required writing a bit of real Java rather than the simplified Processing, so I opted to use the NetBeans IDE which smooths out much of the discomfort in writing java code. While I was at it, I went ahead and redid a bunch of arrays as ArrayLists and HashMaps. (Eclipse is the other major player in free Java IDEs and would probably work just as well for what I used the IDE for.)

The book Visualizing Data by Ben Fry (one of the authors of Processing) was very helpful in explaining how to use an IDE like NetBeans to write Processing sketches.

The TRAER.PHYSICS library for Processing is what handles simulating the springs between similar artists. (Wonderful stuff!)

The data is gotten from the Audioscrobbler web API. Last.fm has launched a new API called Last.fm Web Services, but it requires an account that I didn't want to bother with.

The Source

You can download the source code as a Processing sketch (LastFMGraph.zip). This is probably what you want if you can work with Processing, but haven't gotten into Java yet. This was built using Processing version 0143 and will not work with version 0135. Somewhere between those two versions, they started letting you use Java 1.5 features like ArrayLists that I'm using in this sketch. The Processing version doesn't include the code for viewing multiple users or for fetching pages in a background thread.

You can also download the NetBeans project folder (LastFMGraphNB.zip). I used NetBeans 6.5. This version uses SwingWorker to get the artist similarity data in a new thread. This lets the animation continue to run while pages are being fetched, but it makes the code a little more complicated, so I've left it out of the Processing sketch source.

The version of the sketch that is running as an applet on this page is yet another version that was written to get a set of data that's baked right into the applet. Unsigned applets are only allowed to talk to the servers that host them, so the applet on my site is unable to get data from Last.fm. This is why you have to download the application to be able to see other Last.fm users.

The NetBeans version and the applet were updated in December 2008 to use the 1.01 version of Processing for a pretty significant speed boost.

The Structure of the Code

A computer scientist in a forgiving mood might say that I followed the model, view, controller pattern (MVC).

LastFMGraph.java is where everything gets drawn. This is where most all of the Processing code is including setup() and draw() which are the main parts of this type of Processing sketch. This would be the view.

ArtistGraph.java would be the controller. It keeps track of all the information about who listened to what when and how they are similar. It's also where all the information on the physics simulation is kept.

AudioScrobblerModel.java is the model. It handles fetching XML pages from Audioscrobbler's web interface and parses the XML into lists of other objects like Node.java objects for artists and Friend.java objects for neighbors. These other objects follow what's called a bean pattern where the data in them is accessed via get() and set() methods. This allows for a layer of abstraction that can come in very handy when I want to change how the file name for a weekly artist chart is calculated.

Notes & What's Next

If you look at the artist similarity data returned by Last.fm, it often seems not so good. The graph though is surprising in how sane it seems overall. It's rare to find a dot in a place that makes no sense at all. All the jazz is in the jazz section where the blues are on one side and cool jazz is on the other. The JPop is connected to the main blob of pop about where I would expect it to be. I often find effects like this where the whole of a data set is 'smarter' than you would expect based on individual bits of data. It's sort of like the warehouse full of monkeys at typewriters that can consistently turn out Shakespeare sonnets.

The really big graphs could probably benefit from some sort of summarization. Last.fm has data on what artists are tagged as. It should be possible to get the tag data and check if there's a blob of stuff tagged 'rap' and collapse it into a single circle?

Fuzzy Link Bot

- a network graphing tool based on things that appear near each other in music news articles.

Jones - Music vs the Dow
Jones screen shot1

Jones takes the idea of mapping similar artists and tries to add another dimension (stock market numbers). I think its a bit of a null result, as there is no obvious conclusion that leaps out, but it was still fun to do and an interesting set of technical challenges. It uses the Jung graphing library for graph layout which is a much more robust method than the physics model used in Last.fm graph.

Last.fm Spring Graph is featured in Visualization Vol 2 - Circles - a groovy magazine by the guys at Visual Think Map.