Posing an EoD SQL Question

So here’s the question (and first post in a very long time):

Who would like to see EoD SQL opened up even more?

EoD SQL at the moment is open-source, but even an open-source project is in a way: closed. To add new functionality to EoD SQL you need to climb into the source code. So here’s the idea:

A new MethodImplementationFactory interface that you can implement to provide functionality for a single type of method that could exist in a Query interface. It would look something like this:

public interface MethodImplementationFactory {
    public Class<? extends Annotation> getAnnotationType();

    public void validate(Method method) throws InvalidQueryException, InvalidDataTypeException;

    public MethodImplementation create(QueryImplementation implementation, Method method);
}

The MethodImplementation itself will be an interface (in a way) a lot like java.lang.reflect.Invocationhandler, as it will be the object called to fulfil a method invocation on the implementation object. The MethodImplementation will be provided with plenty of support methods for handling Connection objects correctly and so on, making development of new method implementations much easier.

Why make this change?

This change would allow for easy development of new Annotation instructions, for example: @BatchUpdate or @Call (both requested features). The new instructions could be mixed in with existing annotations such as @Select and @Update without any additional code.

Any thoughts on this? Comments are welcome!

EoD SQL 1.1 Released

EoD SQL 1.1 was released this morning. There are no major new features, but it’s well worth the download.

Included in the release:

• Updatable DataSet objects
• GeneratedKeys.RETURNED_KEYS_FIRST_COLUMN to work around a bug in Derby, or improve performance
• SPI query-generators (only available under Java 6)
• Improved DataSet documentation
• Lots of internal cleaning up
• Several small bug fixes

Feature Requests for EoD SQL

I’ve had a few (ie: 2) requests and inquiries for new features on EoD SQL. Here is the list so far:

1. The optional ability to use JDBC batch functionality (optional because not all drivers support it)
2. A way to use stored procedures from your query interface (probably a new annotation)

I will be working on these features soon (hopefully), although I’d like some input on the first one. How should this be implemented? You don’t want to force batching on a specific query. One option I thought of is a new “BatchUpdate” return type:

public interface MyQuery extends BaseQuery {
    @Update(sql="INSERT INTO users (name, password) VALUES(?{1.name}, ?{1.hashedPassword})",keys=GeneratedKeys.RETURNED_KEYS_COLUMNS_SPECIFIED)
    public BatchUpdate<User> createUser(User user);
}

Which you could then use as:

MyQuery query = QueryTool.getQuery(MyQuery.class);
BatchUpdate<Update> update = query.createUser(null);
query = update.getQuery();

for(User u : users) {
    query.createUser(u);
}

query = update.commit();

getQuery() acts a bit like a “Stack.push()” operation, while commit() acts like the “pop()”. There are loads of other possibilities on implementation here. Haven’t really gone into how the keys will work here, but you can imagine. Let me know what you think!

If you have a feature request, or if you are using EoD SQL for anything: comment! I’m always looking for feedback.

Why GWT is here to stay

I’ve been developing with plain JavaScript for several years now, long before the AJAX craze started (when my spell-checker posted a hidden iframe back to the server and such). Why would I be an advocate of GWT when I’ve written web-apps that include over 10′000 lines of JavaScript? Lets think of this a slightly different way: Why wouldn’t I be an advocate of GWT when I’ve had to maintain 10′000 lines of JavaScript?

Some of the common misconceptions about GWT:

• GWT is for people who don’t want to or can’t code JavaScript
• GWT is only there to hide browser differences (and there are plenty of JavaScript API’s that do this)
• GWT produces bloated JavaScript files
• GWT doesn’t let a graphic designer produce the User Interface
• GWT takes away the raw power of JavaScript
• GWT is a plain Java to JavaScript compiler

I see these complaints everywhere. Those who make them are mostly PHP developers, or people who have never worked on a commercial browser based system that requires large amounts of JavaScript.

At this point, let me say something about GWT that I don’t see explicitly said often enough:

GWT is a Web Application framework!

GWT’s space is not in putting an auto-completing text-input on an otherwise fairly static HTML page. There are plenty of JavaScript API’s that will do that for you. GWT is not “Swing for the browser” either! GWT is more like the Java Desktop Application Framework, or the Netbeans Platform. GWT is not about simply building a user-interface, it’s about reducing the load on your server.

A serious flaw in web frameworks like Struts or WebWork is the amount of data they store in the session. The unfortunate fact is that storing data in your session is very expensive for your server cluster. It’s fine running on a single machine; or if you have very intelligent load balancers, however: running on a larger cluster, or trying to handle tens-of-thousands of simultaneous users becomes really hard.

Why is the session expensive?

In Java you have wonderful Serialization of objects. References are correctly rebuilt when loading a serialized object and so on. Using an HttpSession in a clustered environment however, means that every object in the Session is re-written across the network at the end of every request. Why not just the objects that have changed? The answer is simple: how do you know which have changed?

Struts for example stores all of the Action objects in the Session. Thats potentially a lot of objects to store, and move around between servers. How does this relate to GWT though? In GWT you can place your “index” page behind a standard Filter or a <security> mapping in your Web XML to handle the login of your users, and have GWT work through a service style system, where your servers act in a stateless manor. This drastically reduces the load on your servers, since they no longer need to hold session data in memory or synchronize it with the other servers at the end of each request. In fact, the GWT application can cache the resulting data on the users browser instead of in the Session!

But what about my Graphic Designer?

When developing a desktop application, you generally have a user interface designer working with you (if you don’t, go hire one). Like I’ve already said: GWT is for building applications that happen to run inside a web browser. If you are building a web site, you should probably look at a different technology (though keep GWT in mind all the same). A user interface designer is often not a developer, and so can’t actually write the user interface.

The other side of this argument is: what is your web designer doing writing your HTML?!?!?! In my experience,the web designer produces an image (often in PhotoShop) of the look & feel for the site, and then provides you with the graphical elements you need as you write the HTML for the site. Having a web designer produce HTML is always a bad idea, no matter how much they know about it. If nothing else: HTML limits their creativity.

But I know plenty of JavaScript, why hide it behind Java?

A note to those with this argument:

• Get off your high horse and get back to work

JavaScript is a very powerful language, GWT is not hiding it away behind Java. If you need to code JavaScript in your GWT application: use JSNI, thats what it’s there for. The problem with AJAX is not JavaScript, it’s the way it’s implemented across different browsers. You eventually wind up with loads of “if(browser == IE)” and such all the way through your JavaScript. I won’t even bother to mention the nightmare that is passing events in JavaScript. GWT neatly deals with all of that for you, without any browser checks in the produced code (well, there is one to decide which version of the application to load).

Summary

GWT does not cripple you by having you code Java, it empowers you to do more with less code. The amount of code produced is often smaller than the pure JavaScript API’s that hide browser differences. It’s also a lot easier to work with.

You spend more time working on your application logic and less time re-inventing the wheel.

Development on EoD SQL Continues

A few days ago I released EoD SQL 1.0, and moved the project to Java.net. So now that 1.0 is out the door, where to next? There are several developments that will be taking place in EoD SQL land, and I’m looking for:

• People interested in developing on the codebase
• Input!
  • Bug Reports
  • Feature Requests
  • Ideas
  • What do you use EoD SQL for?

As of now, the next set of developments on EoD SQL include:

1. Writable DataSet objects
2. Internal optimizations
   1. Reuse PreparedStatements
3. Batch Processing (using the JDBC addBatch() methods)
   1. This will be an optional operation in order to cater for JDBC drivers that don’t support batches
4. Custom QueryFactory objects
   1. Based on the ServliceLoader class
   2. Only available for Java 1.6 or higher
   3. Those running 1.5 will not be able to use Custom QueryFactories
   4. Will be the base for an EoD SQL Query compiler

Let me know what you think of the upcoming feature set, what other features would you like to see, and if you’re interested in working on the EoD SQL project.

Windows; Linux and Memory

You often hear how much Linux like memory, and how much it eats up compared to Windows. It’s true! Linux eats into your memory, no matter how much you have, you never seem to have any free for applications to run in! I recently read an article where the author was complaining how Firefox ate into his memory under Vista. The sad fact of it is:

While Linux appears to eat more of your memory, it manages it much better than Windows does.

I have Firefox open right now, I have 10 tabs open with graphics intensive web sites. Firefox is claiming 200Mb of my precious memory… or is it? Actually no: real size in memory 90Mb.

The current list of the most memory intensive applications I’m running, and their “supposed” memory usage:

• Java (Netbeans) 600Mb
• Xorg 380Mb
• Firefox 200Mb
• Thunderbird 130Mb
• Amarok 120Mb
• Kwin (KDE) 120Mb

Now considering I only have 1.5Gb of memory in this machine that 50Mb to much, and considering there are a total of 120 processes running, I haven’t even started to cover the list. Now as any knowledgeable person will tell you, Linux is using some of the hard-drive and pretending that it’s memory, but since a hard-drive is slow, my computer will be running awfully slow.

Wrong! Yes, a disk is much much much slower than memory (especially a laptop drive like this one). However, my computer is running very very fast. Why? Linux’s memory manager notes what parts of memory are used frequently, and what parts are seldom accessed. This means that the parts that are used frequently stay in real memory, and the stuff no-one is using gets stored on the disk.

Translation: Linux will run much faster, and keeps things in memory in such a way as to speed things up! Under Windows, more free memory is better, under Linux it’s the opposite! Less free memory is better, since it means Linux is doing it’s job better.

EoD SQL 1.0 is Finally Here!!!

And the project has moved to eodsql.dev.java.net. New features in the 1.0 release include:

• Added the QueryTool.select method for less structured / once-off queries
• Added the DataSetCache, ArrayDataSetCache and NullDataSetCache for controlling the way a DataSet caches row values
• ConnectedDataSet’s don’t disconnect as often anymore, but are not thread-safe
• DataSourceQueryHandler’s now understand threading, and will reuse open connections if it’s safe to
• The deprecated methods in QueryTool have been removed!
• Brand new tutorial!

Go to the releases folder under Documents & Files to download the latest version!

ReadWriteLock’s on Collections

Many people don’t realize that Collections.synchronized* causes an exclusive monitor lock to be held in every method called on the specified Collection object. For example:

final List<Integer> list = Collections.synchronizedList(new ArrayList<Integer>());

Thread th1 = new Thread(new Runnable() {
	public void run() {
		for(int i = 0; i < Integer.MAX_VALUE; i++) {
			list.add(Integer.valueOf(i));
		}
	}
});

Runnable reader = new Runnable() {
	public void run() {
		for(int i = 0; i < Integer.MAX_VALUE; i++) {
			list.get(i);
		}
	}
};

Thread th2 = new Thread(reader);
Thread th3 = new Thread(reader);
Thread th4 = new Thread(reader);
th1.start();
th2.start();
th3.start();
th4.start();

I am well aware that the code for reading is very bad, but it serves the point. Four Threads, one writing to a synchronized Collection, the others reading from it. This as such is not a problem: writing to a Collection should only ever be done by one thread at a time, while no other threads are reading. However, any number of threads can read from the Collection at the same time. Collections.synchronized* acquires an exclusive lock for both read and write methods, so: only one thread will be allowed to access the Collection at a time.

The solution: Use a ReentrantReadWriteLock, which allows any number of Read locks at a time, but only one Write lock (and no Read locks while a Write lock is held).

There is an even more efficient way to make a Collection thread safe: use an inherently thread-safe collection. Just take a look in java.util.concurrent, and you’ll see that Java comes packed with plenty of them. You will need to decide which method is preferred for what you are doing. Take into account:

• How many reads vs. how many writes
• How large is the collection
• How fast does the code need to execute
• How often will the collection actually be accessed

Why Sony PSP needs Java

The PSP is awesome. I pre-ordered mine, and it still amazes me every time I play a game on it. I’ve been very picky with the games I’ve bought for it (only 4 to date… and considering the time I’ve had it, you’d expect a much larger collection… specially if you looked next to my PS2).

The whole home brew situation on the PSP had me laughing a bit at first. Especially since most of the code written was focused on ripping the games off the UMD and onto the internet. BORING!!! Homebrew is supposed to be about writing new content… not pirating[*] old content.

The PSP is not lacking for First-Party games. But there are times where I’d love to be able to download a Tetris clone for it; or Pacman; or some arb game that someone made up in their spare time. The worry is: opening up the platform could lead to losses for First-Party and paying development houses. The solution: Java.

Why? Because you can sandbox the software. A MIDP VM need have no access of any sort to the underlying file system or hardware. All the PSP needs to allow for a flurry of Java developer activity is:

• MIDP 2
• 3D Extensions
• Wireless Toolkit (for WiFI gaming)

The fact is:

• if people played free games on their PSP…
• They’d be using the device more.
• If they used it more, they’d be buying more games for it.

Java for PSP doesn’t spell the doom on First-Party titles, it’s the best way to sell more titles![*] To quote my brother: I am not a pirate, I am a preemptive maritime salvage expert.

On Netbeans; memory and speed

It’s about the most common complaint about Netbeans: It needs to much memory! *wine wine wine*
That, and: it’s so slows *cry cry cry*

You know something, there are two simple ways around this “problem”:

1. If you are running Windos, switch to a real operating system! I recommend:
   1. Kubuntu for people with permanent internet or
   2. SUSE for those who are on dial-up
2. If you are actually writing code, you need a minimum of 1Gb (preferably 1.5Gb) of RAM
   1. This applies to all software developers, regardless of you language or target platform (I included embedded developers in this statement).
   2. Software developers simply tend to have more software running at any given time
3. A dual-core CPU does make an enormous difference to IDE performance

I do 100% of my development on a 1.6Ghz Centrino laptop, with 1.5Gb of RAM (533Mhz if you must know), with a standard laptop hard-drive. With this simple configuration I am amazed at the speed when I run:

• Netbeans 5.5
• Netbeans 6 beta-2
• JBoss 4
• Thunderbird
• Firefox
• Kopete
• Konqueror
• Amarok
• Kerry-Beagle
• Many terminals
• MySQL

Spread out over 4 KDE desktops, and they all perform as if they were running by themselves. If you have the opertunity, switch to Netbeans 6, as it’s a much faster system than any previous version.