Wednesday, October 29, 2008

Google Developer Day 2008 - lessons learned

Google Developer Day (GDD) 2008 in Prague was very interesting. I focused on scalability and cloud computing sessions and I also managed to visit some Google Map and geo web sessions.

I had a chance to talk to Peter Kukol after his high level presentation and I was really interested if Google uses MapReduce (MR) for all of its massive data processing calculations and it seems that the answer is yes (more specifically the answer is that it is used a lot more often then before and the number of MR jobs is still growing). The interesting point is that there isn't any specific algorithm for which they would consider utilization of different (non MR) architecture. As far as I understand it can be challenging to operate calculation on top of some specific data structures like graphs or trees in MR but it seems that Google has enough resources and can spend some extra cycles of its data centers on non optimized calculations. When it comes to contorl and optimization of MR jobs then the developers can use a lot of visual tools (it would be nice to know more about this).

Also it seems that Google App Engine does not provide support for long running background MR processes now and if it will in the future then there is a little chance that it would directly integrate with Hadoop implementation. This is a pitty because I can imagine that this would allow developers to switch between App Engine and Amazon or other cloud provider very easily.

Anyway, the service in the Clarion hotel was excellent the the meal was delitions! Thanks Google, I hope to be there the next year. All the presentations were recorded on a video and should be available on the youtube in the future.

Some photos:

Opening speach by Jan Šedivý

All the speakers

Relax room exhibition by 3rojka

After party "U Vejvodů"

No comments: