Thursday, October 30, 2008

New Business Product Idea for Sun

As a Java developer I really like what Sun is doing for me (and the list would be very long). I periodically regret to read stories about Sun's poor stock market results. One example of such news can be found here: Sun reports 1.7$ billion loss and falling sales.

I can hardly come up with perfect business plan for them but I have one idea though. Why they don't build a public cluster with Hadoop for rent? As far as I know Sun engineers are interested in Hadoop technology and cloud computing is a new mantra of many web services. If I were to build a new scalable service then I would probably end up using Hadoop at some point as well. The problem is that there is a shortage of public Hadoop compatible clouds (Amazon is probably the only option as of now, Google App Engine does not provide integration with Hadoop now) and Sun has all what's needed.

Wednesday, October 29, 2008

Google Developer Day 2008 - lessons learned

Google Developer Day (GDD) 2008 in Prague was very interesting. I focused on scalability and cloud computing sessions and I also managed to visit some Google Map and geo web sessions.

I had a chance to talk to Peter Kukol after his high level presentation and I was really interested if Google uses MapReduce (MR) for all of its massive data processing calculations and it seems that the answer is yes (more specifically the answer is that it is used a lot more often then before and the number of MR jobs is still growing). The interesting point is that there isn't any specific algorithm for which they would consider utilization of different (non MR) architecture. As far as I understand it can be challenging to operate calculation on top of some specific data structures like graphs or trees in MR but it seems that Google has enough resources and can spend some extra cycles of its data centers on non optimized calculations. When it comes to contorl and optimization of MR jobs then the developers can use a lot of visual tools (it would be nice to know more about this).

Also it seems that Google App Engine does not provide support for long running background MR processes now and if it will in the future then there is a little chance that it would directly integrate with Hadoop implementation. This is a pitty because I can imagine that this would allow developers to switch between App Engine and Amazon or other cloud provider very easily.

Anyway, the service in the Clarion hotel was excellent the the meal was delitions! Thanks Google, I hope to be there the next year. All the presentations were recorded on a video and should be available on the youtube in the future.

Some photos:

Opening speach by Jan Šedivý

All the speakers

Relax room exhibition by 3rojka

After party "U Vejvodů"

Thursday, October 23, 2008

Google Developer Day 2008

I am going to attend Google Developer Day 2008 in Prague tomorrow (feel free to drop me an email [lukas dot vlcek at gmail dot com] in case you would like to meet me). Agenda is full of interesting stuff: Maps and Earth API, App Engine, Open Social Web, GWT, Gadgets... etc. It is clear that I won't be able to track all sessions but I am specifically looking forwar to Peter Kukol's session about large-scale computing architecture including Hadoop and HDFS.

Tuesday, October 21, 2008

Rich Client Programming - NetBeans Platform

I like the NetBeans platform and I have been using it for rich client application development for more then year now. When it comes to learning NetBeans platform API I can recommend book called Rich Client Programming, plugging into the NetBeans platform. I had a chance to study this book quite shortly after it had been released and I also had a pleasure to attend an excellent NetBeans training course held in Prague on 6-7, March. Quite recently I also won this book in NetBeans Puzzler. Folks in SUN are not only giving a lot of high quality SW out for free but they are also doing an outstanding job when it comes to education. Thanks!

I hope to post more about my experience with NetBeans platform development in the future.

Other useful NetBeans platform resources:
Podcasts: http://www.netbeans.tv/podcasts/
API: http://bits.netbeans.org/dev/javadoc/index.html
Geertjan's blog: http://blogs.sun.com/geertjan/

Tuesday, October 14, 2008

Hadoop User Group UK Meeting

Hadoop User Group (HUG for short) was held on August 19th, 2008 in the UK. You can find short video presentations and slides here: http://skillsmatter.com/event/java-jee/hadoop-user-group-meeting

Found via Last.fm blog entry.

Wednesday, October 8, 2008

Maven 2 - Easy Dependency Management for Production

In this post I will cover basic examples of the following maven 2 plugins:
  • maven-dependency-plugin
  • maven-jar-plugin
  • maven-assembly-plugin.
I have been using Maven 1.x for a long time. Now, I am more working with MVN (aka Maven 2) and I found that with MVN it is very easy to prepare distribution package including all dependencies. More specifically, I found two nice approaches how to handle CLASSPATH dependencies during mvn package phase.

The first approach:
maven-dependency-plugin and maven-jar-plugin

As a result of utilization of these two plugins you can have your application classes packed into a jar file and a lib folder next to it which contains all the needed jar libraries. Moreover, the application jar will contain references to all the jar libraries in the manifest. This means that you don't have to put libraries on the CLASSPATH explicitely as long as they are kept in the lib folder.
<plugin>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>copy-dependencies</id>
<phase>package</phase>
<goals>
<goal>copy-dependencies</goal>
</goals>
<configuration>
<outputDirectory>
${project.build.directory}/release-${project.version}/lib
</outputDirectory>
<includeScope>runtime</includeScope>
<overWriteIfNewer>true</overWriteIfNewer>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<artifactId>maven-jar-plugin</artifactId>
<configuration>
<archive>
<manifest>
<mainClass>com.yourcompany.ClasswithStaticMainMethod</mainClass>
<addClasspath>true</addClasspath>
<classpathPrefix>lib</classpathPrefix>
</manifest>
</archive>
</configuration>
</plugin>

The second approach:
maven-assembly-plugin

This plugin allows you to include any kind of resource into the application jar. In case of third-party libraries it means that it un-zips all dependent jar archives and copy compiled classes next to your application classes. Then it zips everything into a single jar file.
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
<archive>
<manifest>
<mainClass>com.yourcompany.ClasswithStaticMainMethod</mainClass>
</manifest>
</archive>
</configuration>
<executions>
<execution>
<id>make-assembly</id>
<phase>package</phase>
<goals>
<goal>attached</goal>
</goals>
</execution>
</executions>
</plugin>


Related resources:
Original mail thread from maven mail archive: here