Monday, August 27, 2007

Is it possible to create SW that grooves like this?

Let's take this fantastic music video from youtube:

Billy Cobham on drums.
Highly optimized fast relation database running on a cluster of two independent servers (double kick drum). Experiencing indices maintenance and optimizations from 5:24 to 6:11. All other SW components (keyboards and bass) are rather idle at this time. You can see that tuning of database is a very hard task. Every piece (drum) must be hit at least once or twice for better. Only few can do it well and only real masters can do it with their eyes closed.

George Duke is playing keyboards.
Fast and light Perl based web UI. Only a real poet mind can use Perl effectively like that with such smile on the face while the same thing can be expressed in such many ways (at least 4 keyboards each with plenty of options). We can see that relatively simple commands in this layer can cause dramatic response in the DB layer (4:32-5:19).

Larry Kimpel drives the bass.
The glue between the Perl and DB is a system kernel with simple, yet very effective command line interface. Just couple of commands can do a lot for you but one has to be very careful about order of individual commands, options, pipes and arguments. Cron daemon is your good friend when it comes to repeating calls in ridiculous frequency (like 19/16 or so...)

As for boot up time:
You can see that the most expensive service is the DB layer. Perl starts very fast and the command line is just present (it seems as if it does not require any time for initialization at all).

As for releasing the resources: (simply the end of process)
Again we can see that the DB takes its time. But we can see that both the Perl and kernel layers wait until it finishes properly (good practice).

This must be some real time application (internet service?) or heavy I/O involving calculation (indexing of database content?). But somehow I feel that this music session grooves one (or two) steps further then any of SW compilations I have ever seen...

Sunday, August 19, 2007

Machine Learning Databases Repository

UCI Machine Learning Repository is a collection of 160 databases which can be used for machine learning algorithms. It was started in 1987 and now a new web interface with search functionality is being tested.

Saturday, August 18, 2007

Scalability and Efficiency of Data Mining

There is a nice video presentation about scalability and efficiency aspects of parallel computation. It touches MapReduce paradigm and a wide portion of the presentation is devoted to a classical problem called Frequent Itemset Mining. Experimental results of other classical data mining tasks are presented as well.

Interestingly, Doug Cutting (one of the leading developer of Hadoop) have a post on his blog about how to use MapReduce to implement ten different machine learning algorithms.

If I understood correctly one of the main points of Wagner's presentation is that current MapReduce is great for stateless computations but it can be a problem (less effective) when stateful approach is needed. For their needs they created MapReduce derived implementation where each Reduce phase can store results and other metadata into external repository so that other tasks can learn about it very fast (so that subsequent Map task can start earlier if it has all the information it needs and does not have to wait until the whole Map phase finishes).

Friday, August 17, 2007

Do You Want To Live Forever?

I have just finished watching this video. It makes me think what can happen if computer developer gives up the job...

(This video is not a joke... ok, may be I should have said it is not a fake. That is more convenient formulation I think.)

Sunday, August 5, 2007

The most interesting world's cities in VR

The project called 360cities an interactive collection of panoramic photos from some of the most interesting cities in the world. It also contains a lot of beautiful photos from Praha. It seems to be built on top of Google Maps technology and not only it can help you when planning vacation but can be also interesting tool if you want to rent or buy an apartment. They say the city collection is growing fast so be sure to check back soon.

Thanks to my brother for this interesting link.