EoD SQL just got a 2.1 Release Candidate kicked out the door. While there are (once again) many performance improvements and a few bug fixes, the big news of this release has got to be Batch Updates!

Some people (myself included) have been waiting a very long time to see this feature in EoD SQL, and it’s taken a fair amount of discussion to decide how it would work. The final answer came from Bernd Rinn: allow parameters to be specified as various Collection types. So to add a list of User objects to your database:

@Update(sql="INSERT INTO users (name, birth_date, password, email) VALUES (?{1.name}, ?{1.birth_date}, ?{1.password}, ?{1.email})",batchUpdate=true)
void insertUsers(List<User> users);

Batch updates will work with any class extending Collection, or an array type. It can also be used with simple parameters:

@Update(sql="REPLACE INTO user_group_join (user_id, group_id) VALUES (?{1.id}, ?{2.id})",batchUpdate=true)
void ensureUserInGroups(User user, Collection<Group> groups);

A Batch Update is generally much faster than a series of normal update invocations, because EoD SQL will use PreparedStatement.addBatch() to build up all of the parameters on the client-side before sending all of the data to the database in one statement. It doesn’t mean it’s a single transaction, but it does mean that the database can execute all of the parameter variations before sending any data back.

So why not go download the new release and give it a try.

Bernd Rinn

EoD SQL 2.1-Beta has been released, you can go download it from the Documents & Files section of the homepage. The 2.1 release is about improving speed and usability over the 2.0 release. Some of the new features included are:

• Specific Collection implementations (including those not in the standard Java API’s) can now be returned:

  @Select("SELECT * FROM users WHERE group = ?{1.id}")
   ArrayList<User> selectUserGroup(Group group);

• Select methods can now populate existing objects:

  @Select(sql="SELECT * FROM user_meta_info WHERE user_id = ?{1.id}",into=1)
  void selectMetaInfo(User user);

• Faster construction of query implementations
• Better error reporting
• Several small bug fixes

This is of course a beta release, so there may be bugs. That said the code is pretty well unit tested and should be relatively stable. One new feature not in the release (but on the way) is batch updates. Stay tuned for more information!

The Aladdin eToken is a great little security device when you have sensitive private key information. They are a great USB solution in that they support all major operating-systems with a PKCS#11 driver. This means they also integrate with Java on all major platforms (through the SunPKCS11 Security Provider).

The one problem we have come across  with the device is trying to create keys on it using the standard Java keytool application. There are two errors you’ll almost certainly face, both in the form of a sun.security.pkcs11.wrapper.PKCS11Exception.

Error 1: CKR_ATTRIBUTE_TYPE_INVALID

This error (we found) pops up when you don’t specify the key algorithm on the command line. You need to specify “-keyalg RSA” to ensure that an RSA key (instead of the default DSA) is generated for the eToken.

Error 2: CKR_TEMPLATE_INCONSISTENT

This one is a little more nasty.  It’s not well documented, but effectively the keytool doesn’t generate signatures that the eToken’s PKCS#11 implementation likes (we couldn’t find any tool other than the eToken software that did). Under Windows you need a registry key:

HKEY_LOCAL_MACHINE\SOFTWARE\Aladdin\eToken\MIDDLEWARE\GENERAL

The key is a DWORD attribute: “TolerantX509Attributes” with a value of “1″. But what about Mac and Linux. You’ll need to look for the eToken config file: /etc/eToken.conf, then under the “[GENERAL]” section of the file add the line:

TolerantX509Attributes=1

This will allow keytool (and other PKCS#11 tools) to generate and store keypairs on your USB eToken.

A Command Line Example

keytool -v -genkeypair -keyalg RSA -alias myKey -keystore NONE -storetype PKCS11 -providerClass sun.security.pkcs11.SunPKCS11 -providerArg /home/me/pkcs11.conf

The Dreaded “Not Serializable” Problem

GWT turns all of your Java code into JavaScript, but it also obfuscates everything. With this in mind, it makes serialization a bit more complex than usual. GWT generates a serializer and deserializer for each class that could be transported across the wire to an RPC service. The difficulty comes in knowing which types had code generated, and which didn’t. GWT solves this problem for itself with the Serialization Policy file, where the compiler lists all of the classes that the client code will know how to deserialize.

This however leads to another problem: what happens when something unexpected gets in the way. Hibernate is the most used, and thus most complained about when it comes to Serialization problems. Hibernate turns any Collection object into a special “lazy” implementation that will only load your data when you ask for it. All very fine and well for a server bound application, but a GWT application needs all that data up front for serialization. When GWT comes to sending that List<Message> to the client, it chokes. It knows how to serialize an ArrayList, and LinkedList (since you worked with them on the client side), but a Hibernate List or PersistentCollection is a totally unknown type, so it’s not in the Serialization Policy, so the server side throws an Exception at you.

So how does EoD SQL help with these problems? Read on to find out!

Loosing the Lazy Loading

EoD SQL  (unlike most OR mapping layers) makes no attempt to understand the internal relationships between you objects. The reason for this is that EoD SQL is technically dialect neutral, it doesn’t actually care what you put into you @Select annotation so long as it can create a PreparedStatement and populate it with parameters. This frees you to take maximum advantage of database extensions, or weird SQL dialects. It also means that EoD SQL will never generate any SQL code for you: what you type is what gets executed.

Within a GWT application, you don’t have a way to block until a method is complete, and thus lazy-loading data becomes a chore, since you need to work with a lot of callbacks (a nice way of abstracting lazy-like-loading is posted on my blog). This means that most of the object model given to GWT needs to be (at least partly) eager-loaded.

So because EoD SQL doesn’t do any lazy-loading, it “forces” you into thinking about how much data and what data you will be sending to the client. It takes a little getting used to, but over time you will find your code much clearer to read and running much faster (since less data comes from the database, fewer queries are run and the JavaScript has less data to deal with).

Arrays instead of Collections

The first trick to working with GWT is: when you need more than one object fetched from the database (which is most of the time), return an array instead of a Collection:

public interface MessageQuery extends BaseQuery {

/*
* This is not a great way to do things in EoD SQL 2.0 when dealing with GWT.
* This method will return an ArrayList of Message objects, however the List implementation could
* possibly be changed in the future (not likely). Further the ArrayList is populated using the basic
* List.add(Object) method. The ArrayList will re-copy it's internal array over-and-over again as
* the ResultSet is scanned.
*
* So while this method will work with GWT, it's really not recommended.
*/
@Select("SELECT * FROM messages WHERE parent_id = ?1 ORDER BY date DESC LIMIT ?3 OFFSET ?2")
List<Message> selectChildMessagesAsList(int parentId, int start, int count);

/*
* This won't work at all with GWT. The DataSet may offer lazy-loading
* and some other wonderful tricks, but since the client code has no idea
* how to deal with a DataSet it's useless here. I thought I'd put it in for completeness.
*/
@Select("SELECT * FROM messages WHERE parent_id = ?1 ORDER BY date DESC LIMIT ?3 OFFSET ?2")
DataSet<Message> selectChildMessagesAsDataSet(int parentId, int start, int count);

/*
* This is the way to do it! Using only arrays for Serialization has a side effect
* on the GWT side of things: less JavaScript code for your clients to download.
* GWT understands how to deserialize arrays, and doesn't need to include
* Serialization code for different List implementations.
*/
@Select("SELECT * FROM messages WHERE parent_id = ?1 ORDER BY date DESC LIMIT ?3 OFFSET ?2")
Message[] selectChildMessagesAsArray(int parentId, int start, int count);

}

Now for the big question: why are arrays “so much better” than a List of some sort? EoD SQL takes fairly extreme measures to keep the performance of an Array as fast as possible. For the most part (that is: on most JDBC drivers) the array will be pre-sized and fetched backwards. This sounds very strange, but it is generally a lot faster than trying to work forwards and re-growing the array whenever it runs out of space (and then down-sizing it at the end). EoD SQL will run forwards if the JDBC driver requires it, but for the most part it runs backwards.

This technique will not only improve performance, JavaScript size, and simplicity of the code (arrays are just easier in most cases); it’ll also reduce the amount of memory your server uses (since the array is pre-sized and never copied).

Finally: because the type is declared as an array, you know exactly what Serialization code GWT is going to generate.

Looking to the Future (EoD SQL 2.1)

In EoD SQL 2.0 you can declare Collection objects as return types, but only as their interfaces (as you saw in the above code example). EoD SQL 2.1 has a new feature which allows you to declare concrete implementations as return types. This is the preferred method of declaration for GWT (since again: it minimizes the amount of JavaScript generated). A few nice little examples:

public interface MessageQuery extends BaseQuery {

@Select("SELECT * FROM messages WHERE parent_id = ?1 ORDER BY date DESC LIMIT ?3 OFFSET ?2")
ArrayList<Message> selectMessages(int parentId, int start, int count);

@Select("SELECT title, date, author FROM messages WHERE parent_id IS NULL ORDER BY date")
MyCollectionImplementation<Message> selectTopLevelMessages();

}

The big difference between 2.0s handling of Collection objects, and 2.1 is that 2.1 will allow the Collection implementation (including those declared as their interfaces) to optimize their fetching of the data internally (if they are able). Thus returning a List or ArrayList will no longer cause the resize-resize-resize effect (or shouldn’t anyways). Collections will be given the same treatment as arrays in 2.1.

Conclusion

So we’ve looked at four different situations where EoD SQL works really well.

The concept to take away from this (I feel) is that: with EoD SQL you are in total control of how the code is going to behave.

If you need lazy-loading and scrolling data (like in our Swing example), you can use a DataSet; to minimize memory a DataIterator; and when lazy-loading can’t be used because of other constraints Collections and Arrays are available. Whatever the situation: EoD SQL is going to make life that little bit easier for you, without you giving up control of what goes on behind the scene.

JavaScript vs. Web Applications

So far in this series we’ve discussed using DataSets for Swing applications and DataIterators for web-applications. Why would I now bring in JavaScript as something outside of “web-application”? JavaScript applications have very different requirements to a normal web-application. Where a normal web-application has little ability to do things like preload data (like the next page), a JavaScript application may (for example) download the entire data-set and then display it in pages. This next section is about binding to JSON for JavaScript applications.

First thing to remember here is that an EoD SQL DataSet is a List and thus compatible with the Collections API. For this example we’re going to be working with the outstanding GSON API from our friends at Google. Our objective here is to minimize the amount of time spent between the Database and pushing the data to the client. Because GSON doesn’t appear to support Iterable object out-of-the-box, we’re going to start off using a DataSet.

Using a DataSet with GSON

The advantage with this approach is that you don’t need any additional binding classes in GSON. The approach should work well with any JSON API. The DataSet class has the additional advantage of being able to perform “paged” requests on databases that don’t have a LIMIT statement (by using absolute get() invocations or DataSet.subList()).

public class UserQuery extends BaseQuery {
@Select(sql="SELECT * FROM users ORDER BY name LIMIT ?2 OFFSET ?1",cache=NullDataSetCache.class)
DataSet<User> selectUserPage(int start, int count);

@Select("SELECT COUNT(*) FROM users")
int getUserCount();
}

public class PageOfUsersServlet extends HttpServlet {

private static final UserQuery USER_QUERY = QueryTool.getQuery(UserQuery.class);

protected void doGet(
final HttpServletRequest request,
final HttpServletResponse response)
throws ServletException,
IOException {

final int start = Integer.parseInt(request.getParameter("start"));
final int count = Integer.parseInt(request.getParameter("count"));

final DataSet<User> users = USER_QUERY.selectUserPage(start, count);

try {
response.setContentType("application/json");
final Gson gson = new Gson();
gson.toJSON(new PageResponse<User>(
start,
USER_QUERY.getUserCount(),
users),
response.getWriter());
} finally {
users.close();
}
}

private static class PageResponse<E> {

private final int startItem;

private final int totalItems;

private final Collection<E> items;

PageResponse(
final int start,
final int total,
final Collection<E> items) {

this.startItem = start;
this.totalItems = total;
this.items = items;
}

public int getStartItemIndex() {
return startItem;
}

public int getTotalItemCount() {
return totalItems;
}

public Collection<E> getItems() {
return items;
}
}
}

Some important things about the above code:

1. We use the NullDataSetCache in the @Select annotation. This stops the returned DataSet caching the objects for identity equality, and since JSON has no back-referencing support, such caching would be a waste of time
2. We wrap the Users returned in a PageResponse object and serialize that to the client. This object allows for a single callback function to handle the responses from any request for a page of users (since the response contains the index to insert the first user in). So we avoid the need for a closure on the JavaScript side.
3. We send the total number of users in the database back to the client with each request, allowing us to determine the number of pages that can be fetched, even when the number of users is constantly changing.
4. Finally: we have GSON pipe the JSON directly to the responses Writer. If the response is small it’ll be buffered by the container and then sent, but if we overflow the buffer boundary, the response will be chunked into blocks allowing us to free up some server-side memory.

DataIterators and Rubberstamping

Now for some black-magic. This part has only been tested with GSON, it may work with other JSON api’s but it strongly depends on how they treat Collection objects. The idea here is to avoid writing special binding objects and to leave the selection of data from the database until as late as possible. We rely on the fact the GSON never uses the size() or toArray() methods of a Collection object, only iterator(). This means that we can have a Collection implementation that doesn’t have a size() method, and returns a DataIterator for it’s iterator() method.

The example is virtually identical to the one above, except that it will run much faster, and use much less memory.

public class UserQuery extends BaseQuery {
@Select(sql="SELECT * FROM users ORDER BY name LIMIT ?2 OFFSET ?1",rubberstamp=true)
DataIterator<User> selectUserPage(int start, int count);

@Select("SELECT COUNT(*) FROM users")
int getUserCount();
}

public class PageOfUsersServlet extends HttpServlet {

private static final UserQuery USER_QUERY = QueryTool.getQuery(UserQuery.class);

protected void doGet(
final HttpServletRequest request,
final HttpServletResponse response)
throws ServletException,
IOException {

final int start = Integer.parseInt(request.getParameter("start"));
final int count = Integer.parseInt(request.getParameter("count"));

final Collection<User> lazyUserCollection = new AbstractCollection<User>() {

public Iterator<User> iterator() {
// we perform the query as late as possible
return USER_QUERY.selectUserPage(start, count);
}

public int size() {
// we could return the "count" field here,
// however the "count" is simply the maximum number
// of rows that could be returned, not the actual number
throw new UnsupportedOperationException();
}

};

response.setContentType("application/json");
final Gson gson = new Gson();
gson.toJSON(new PageResponse<User>(
start,
USER_QUERY.getUserCount(),
lazyUserCollection),
response.getWriter());
}

private static class PageResponse<E> {

private final int startItem;

private final int totalItems;

private final Collection<E> items;

PageResponse(
final int start,
final int total,
final Collection<E> items) {

this.startItem = start;
this.totalItems = total;
this.items = items;
}

public int getStartItemIndex() {
return startItem;
}

public int getTotalItemCount() {
return totalItems;
}

public Collection<E> getItems() {
return items;
}
}
}

Conclusion

We’ve run through using both DataSets and DataIterators for turning your objects into JSON data for a client. Bare in mind that you should test the DataIterator technique well before using it with any other JSON API. In the final part of this series we will look at using EoD SQL with GWT, and take a sneak peak into future of EoD SQL and the upcoming 2.1 releases features.

Static Data Display

In a Swing application with a direct connection to a database, it makes sense to leave the database to do much of the heavy lifting. The Swing client can keep a minimum amount of data in memory, while holding a scrollable Result Set open on the database (fetching more of the data as required). In a web application on the other hand, things are a little more complex. For one thing, your database isn’t exposed to the any part of the network; for another: your clients have no ability to keep Result Sets open; and finally: you generally are working with much higher loads than a Swing application.

EoD SQL has many different approaches to loading data from the database. We’ve already looked at DataSet objects, which remain connected to the database and work well with scrollable, cursor backed, Result Sets. DataSets are (in a way) a leftover from the origional EoD API from Java6-beta (where they were the only way to fetch objects from the database). In this post we’ll take a look at an EoD SQL idea: DataIterators.

Data Iterator Introduction

A DataIterator as the name implies as an Iterator which is connected directly to the database (like a DataSet). Here are the most important features of a DataIterator:

• DataIterators are forward-only (no going backwards)
• They are always connected to the database (no notion of a “disconnected” DataIterator)
• They are not backed by any sort of cache (ie: DataSetCache)
  • There is no need for one, since you can’t go backwards
• They also implement Iterable (their “iterator()” method returns “this”) for convenience
  • So they work easily with the Java-5 enhanced for loop
• They close themselves when their last object has been read (ie: “hasNext()” returns false)
  • If you do not work through all of the DataIterator: you will need to invoke it’s “close” method by hand
• You cannot modify a DataIterator (remove() will throw an UnsupportedOperationException)
  • The following is an error (InvalidQueryException)
    @Select(sql="SELECT * FROM users",readonly=false)
    DataIterator<User> selectUsers();

DataIterators are a great fit whenever you only need to work through the ResultSet once and in a single direction.  The database can stream the ResultSet to the JDBC driver, since there is no need for a scrollable cursor. This forward-only, streaming nature is a very good fit to web-applications. Web applications work with a user that is on the other side of an HTTP connection. So scrollable DataSets that hold onto their database Connections are of limited use, since a web application generally holds onto a database Connection for as short-a-time as possible (while DataSets are best used with long lived Connections).

Rubberstamping

Rubberstamping is the key feature we will be looking at in this post.  Rubberstamping is only available to a DataIterator, and is setup in the @Select annotation:

@Select(sql="SELECT * FROM users",rubberstamp=true)
DataIterator<User> selectUsers();

So what is rubberstamping? Take a look at the following code and you’ll see.

DataIterator<User> users = userQuery.selectUsers();
User user1 = users.next();
User user2 = users.next();

if(user1 == user2) {
  // this is the big deal, this will always be true
}

Rubberstamping only ever creates one “row” object, in this case a single User object is created. Each time you invoke next(), instead of creating a new User object (and so forcing the GC to clean the old one up): it pushes the new values into the exsting User object and returns it. This saves memory and time (both allocation time and GC time later on). You could argue that the eden-space GC will take care of these short lived objects, but in a web environment the rubber-stamped row object will also be handles by the eden-gc and so the eden GC will trigger less often (since there will be fewer objects).

Rubberstamping and Web Applications

So why is rubber-stamping so nice for web applications in particular. A web-application is connected (fairly directly) to a socket that the browser sits on the other end of. Objects being processed for display as HTML are almost never needed (within the same request) after the HTML has been written. In this case: rubber-stamping is a perfect fit, since the object is re-used instead of being discarded.

I find the best way to work with a DataIterator in a web-application is with the JSTL “forEach” tag which supports DataIterator as a Iterator instance. Generally you should try and avoid using the begin and end attributes of the tag, opting instead for a “LIMIT” (or similar) clause in your SQL statement. Because the for each tag uses the DataIterator as a plain Iterator object it fully supports the rubber-stamping ability, and because you are outputting directly to a JspWriter you almost certainly won’t need to keep references to objects already outputted.

Comparison to the JSP SQL tags

So why not just use the JSP SQL tags then? There are several reasons to rather make use of the DataIterator class.

• The JSP remains agnostic to the type being passed in, any object supported by the “forEach” tag can be used, if a DataIterator becomes unsuitable in the future you can swap it without modification to the JSP.
• The SQL code remains in an EoD SQL BaseQuery interface which is (or should be) executed by normal Java code and passed into the JSP using a request attribute (or similar mechanism).
• The SQL query (and thus the DataIterator) is centralized, and so usable by the Java code that supports the JSP. So you avoid code duplication (and wondering why the JSP is still showing the wrong data).
• You’re working with your objects and not just raw SQL data.

Caching Data from the Database

A major concern now is database performance. With a DataIterator you will be accessing the database every time you want to read some data. Many web-application make use of a caching system (such as memcache, terracotta, eh-cache or similar), how can you have a DataIterator interact with the cache? As I already said in the previous section: the “forEach” tag doesn’t care if the input type changes between 2 invocations.

I generally put my JSP pages “behind” a Servlet which invokes the JSP using a RequestDispatcher.forward call. Here’s a simple example of a Servlet that invokes a JSP, and plays nicely with an imaginary “cache” object.

public class TopicListServlet extends HttpServlet {
  private final Cache<Long, Collection<Topic>> cache = ...;
  private final TopicQuery query = QueryTool.getQuery(TopicQuery.class);

  private Iterator<Topic> getTopicsFromCache(final Long id) {
    final Collection<Topic> topics = cache.get(id);
    return topics != null
      ? topics.iterator()
      : null;
  }

  protected void doGet(
      final HttpServletRequest request,
      final HttpServletResponse response)
      throws ServletException,
      IOException {

    final Long id = Long.parseLong(request.getParameter("id"));
    Iterator<Topic> topics = getTopicsFromCache(id);

    if(topics == null) {
      topics = new CachingIterator<Long, Topic>(id, cache, query.selectTopics(id));
    }

    request.setAttribute("topics", topics);
    request.getRequestDispatcher("/topics.jsp").forward(request, response);
  }

  private static class CachingIterator<K, V> implements Iterator<V> {
    private final Iterator<V> delegate;
    private final Cache<K, Collection<V>> cache;
    private final K key;

    private final Collection<V> values = new ArrayList<V>();

    public CachingIterator(
        final K key,
        final Cache<K, Collection<V>> cache,
        final Iterator<V> iterator) {

      // ...
    }

    public boolean hasNext() {
      if(!delegate.hasNext()) {
        cache.add(key, values);
        return false;
      }

      return true;
    }

     public V next() {
       final V value = delegate.next();
       values.add(value);

       return value;
     }

     public void remove() {
       throw new UnsupportedOperationException();
     }
  }

  public static interface TopicQuery extends BaseQuery {
    @Select("SELECT * FROM topics WHERE parent_id = ?1 ORDER BY post_date DESC")
    DataIterator<Topic> selectTopics(long parentId);
  }
}

As you can see in the above code, the need for caching has a potential side-effect: you loose the ability to rubber-stamp the objects. However this is not the end of the world, since rubber-stamping is particularly useful with potentially very large data-sets (where caching is often a bad idea). Another, often easier option than was used here (although slower for your request) is to have EoD SQL return a Collection<Topic> or List<Topic> instead of a DataIterator. This mechanism however is much slower, since EoD SQL will read in the entire ResultSet  before returning from the select query. We’ll explore using Collections as return types in the next two posts.

Conclusion

We’ve played with rubber-stamping, perfect for those days when you need lots of data with minimal memory usage. We’ve also looked at how to get the objects from a DataIterator into a cache for later use. DataIterators are a generally good mix with web applications, since you have a socket open, so you really want to pre-buffer as little data as you possibly can. In the next post we’ll take a look at using EoD SQL with JavaScript applications. While they are web-applications as well, they have different communication requirements to a typical web-application. So we’ll look at applying much the same techniques to AJAX applications.

Swing Applications

Introduction

We start off the series with Swing Applications. Why? Although probably the application type least likely to be using EoD SQL right now, they nevertheless make use of the most commonly demonstrated EoD SQL data-structure: the DataSet. Swing Applications often have direct access to their database. They can therefore make much better use of long-lived database connections and cursor based ResultSets than web-applications can. So we open with a more detailed than-usual discussion of the DataSet interface and it’s implementations.

The DataSet is the original return type of the EoD API from Java 6 – beta. It’s a relatively thin wrapper on top of a scrollable ResultSet, and it performs best when your database that supports cursors, and your client that will be looking at large volumes of data over a longer periods of time. You’ll notice that this fits well with an intranet Swing client that has a direct connection to it’s database.

DataSet

Despite there only being one DataSet interface and one @Select annotation in EoD SQL, there are three DataSet implementations:

1. ConnectedDataSet
2. DisconnectedDataSet
3. UpdatableDataSet

The two we will be working with in Swing will be the ConnectedDataSet and the WritableDataSet (the DisconnectedDataSet effectively acts like a normal List object, and so isn’t interesting in the context of a Swing application).

Now some important factors to remember about the default DataSet implementations in EoD SQL:

• DataSet extends the java.util.List interface and thus can be treated as a normal Java Collection object
• DataSet implementations are not Serializable
• Like the basic Collection implementations in Java SE, DataSet is not thread-safe
• The ConnectedDataSet and UpdatableDataSet hold a Connection open when they are in use
  • These implementations need to be closed when they are no longer in use
• The ConnectedDataSet and UpdatableDataSet both use a DataSetCache to cache objects
  • Most commonly used to keep identity equality working as expected
  • Can also be extended (later in this post) to perform more complex caching of objects

The Swing side of things

Swing has the advantage of being backed by generic Model interfaces, allowing you to back many Swing objects directly against the database through a DataSet object. When backing your Swing application onto a cursor based database (such as Postgres or Oracle), the DataSet class allows you to very quickly fetch data from the database and display it to the user without fetching the entire ResultSet into memory up front. It also allows you to treat the data within the Swing object as it’s true Object type rather than a Database row (as Swing was intended to do). Considering that Swing widgets generally only display a snapshot of all the available data (JList, JTree, JTable are all scrollable) working with a DataSet means that only the displayed data needs to be in memory (and sometimes not even that data).

Example: A ListModel

public class Movie {
 @AutoGeneratedKeys
 public long id;

 public String title;
 public String director;

 // ...

 public String toString() {
  return title;
 }
}

public interface MovieQuery extends BaseQuery {
 public static final MovieQuery MOVIES = QueryTool.getQuery(MovieQuery.class);

 @Select("SELECT * FROM movies ORDER BY title")
 DataSet<Movie> selectMovies();

 // ...
}

public class MovieListModel extends AbstractListModel {
 private final DataSet<Movie> movies = MovieQuery.MOVIES.selectMovies();

 public MovieListModel() {
 }

 public int getSize() {
  return movies.size();
 }

 public Object getElementAt(final int index) {
  return movies.get(index);
 }

 /**
 * Unlike a normal ListModel this implementation will hold open a
 * database Connection until we release it. Thus when the window
 * our JList is added to gets closed, we should get closed as well,
 * unfortunately for this example: we have no knowledge of when the
 * window will be closed, and as such we rely on outside code to
 * invoke our close() method so that we can close our DataSet.
 */
 public void close() {
  movies.close();
 }
}

DataSetCache

Every ConnectedDataSet and UpdatableDataSet makes use of a DataSetCache object (each DataSet instance has it’s own DataSetCache instance). The default DataSetCache returned by a methods annotated with @Select will use WeakReference objects to allow the Garbage Collector to destroy any objects no longer in use. The main purpose of this default implementation is to keep identity equality working. Since the objects are only fetched from the DataSet during painting and when the outside application asks for the selected objects, the Garbage Collector is free to throw away any objects that are not visible (or even those that are visible, but have already been painted). The following code illustrates the difference between the ArrayDataSetCache (default) and the NullDataSetCache (which does no caching):

public interface MovieQuery extends BaseQuery {
  @Select(sql="SELECT * FROM movies LIMIT 2",cache=ArrayDataSetCache.class)
  DataSet<Movie> selectMoviesWithCache();

  @Select(sql="SELECT * FROM movies LIMIT 2",cache=NullDataSetCache.class)
  DataSet<Movie> selectMoviesWithNullCache();
}

public class MovieQueryTest {

  @Test
  public void testCacheIdentity() {
    final MovieQuery query = QueryTool.getQuery(MovieQuery.class);
    final DataSet<Movie> ds = query.selectMoviesWithCache();

    final Movie m1 = ds.get(0);
    final Movie m2 = ds.get(1);

    assertNotNull(m1);
    assertNotNull(m2);

    assertNotSame(m1, m2); // m1 != m2

    assertSame(m1, ds.get(0)); // m1 == ds.get(0)
    assertSame(m2, ds.get(1)); // m2 == ds.get(1)
  }

  @Test
  public void testNonCacheIdentity() {
    final MovieQuery query = QueryTool.getQuery(MovieQuery.class);
    final DataSet<Movie> ds = query.selectMoviesWithNullCache();

    final Movie m1 = ds.get(0);
    final Movie m2 = ds.get(1);

    assertNotNull(m1);
    assertNotNull(m2);

    assertNotSame(m1, m2); // m1 != m2

    // now the interesting part
    assertNotSame(m1, ds.get(0));
    assertNotSame(m2, ds.get(1));

    // and finally, just to prove the point
    assertNotSame(ds.get(0), ds.get(0));
  }
}

The problem is: this can create a lot of load on your database when you have tens-of-thousands of concurrent users (Swing applications seldom see the scale of concurrent users to the database that web applications do, which is why a ConnectedDataSet still makes sense). In this case it’s a good idea to change how the objects are cached by the DataSet by implementing your own custom DataSetCache (such as a SoftReference cache or a Least Recently Used (LRU) cache). This will allow you to make fewer round-trips to the database, while still taking advantage of a cursor based database. A very simple example of a Least Recently Used cache based on a LinkedHashMap might look as follows:

public class LRUDataSetCache<T> implements DataSetCache<T> {
 private final Map<Integer, T> map = new LinkedHashMap<Integer, T>(16, 0.75f, true) {
  protected boolean removeEldestEntry(final Map.Entry<Integer, T> entry) {
   return size() > 64;
  }
 };

 public void init(final DataSet<T> dataSet) {}

 public boolean isCached(final int row) {
  return map.containsKey(Integer.valueOf(row));
 }

 public T getObject(final int row) {
  return map.get(Integer.valueOf(row));
 }

 public void setObject(final int row, final T object) {
  map.put(Integer.valueOf(row), object);
 }

 public void destroy() {
  map.clear();
 }
}

This is not a very good or efficient way to cache rows in an LRU mechanism, but it’s a simple example. The only rule for a DataSetCache (besides implementing  the DataSetCache interface) is that it must have a default constructor (public with no parameters). The next question is, how do we integrate the new cache with our application? The same way as we did with our unit test above:

@Select(sql="SELECT * FROM movies ORDER BY title",cache=LRUDataSetCache.class)
DataSet<Movie> selectMovies();

The result: our MovieListModel will now keep up to 64 of the most recently used Movie objects in memory for the DataSet, eliminating the need to re-fetch them from the database. Note: we haven’t dealt with time-to-live or any such common caching practices here, it’s way beyond the scope of this post (however, you may want to look into doing such things if you are implementing a more complex DataSetCache).

UpdatableDataSet

So we’ve covered the reading side of a DataSet: opening a DataSet; closing a DataSet; and caching the data within a DataSetCache. What about writing data back into a DataSet? The ResultSet interface has several methods to add new rows and update or delete existing rows. This code is available to you through EoD SQL, but is subject to the same constraints as those placed on a ResultSet object by the JDBC spec and driver. Some databases do not support updatable ResultSets at all, in which case this functionality will not work for you (check the “database_notes.txt” file in the docs directory of your EoD SQL distribution).

The most common constraints placed on an updatable ResultSet are:

• You must only select from a single table
• You must select the entire primary-key
• You generally must select all of the columns that don’t have a default value (and populate them)

There is also one very important note with an UpdatableDataSet: it will not pickup changes to your objects unless you explicitly invoke the set method for that row and object. The main reason for this is that EoD SQL does no enhancement of your bytecode (in fact EoD SQL is quite capable of running in a very tight sandbox), and as such has no way of knowing when your objects change (and then no way of knowing when you want those changes pushed back into the database). Thus you need to tell it when you’ve updated the object by “putting it back” into the DataSet.

UpdatableDataSet Mutation Methods

UpdatableDataSet actually extends the default ConnectedDataSet class. ConnectedDataSet has no mutation methods available (add, remove, clear and co all throw UnsupportedOperationExceptions). UpdatableDataSet implements some, but not all of the mutation methods defined in the List class:

• add(E)
• addAll(Collection<? extends E>)
• remove(int)
• set(int, E)

But what about all the other mutation methods like add(int, E) or retainAll(Collection<?>)? These methods has been deliberately excluded from the implementation for some important reasons:

1. The ResultSet update methods don’t allow inserting rows at specific indicis. Thus add(int, E) and addAll(int, Collection<? extends E>) are impossible
2. Methods like remove(Object) and retainAll(Collection<?>) require a scan of the entire DataSet to find which Object or Objects to remove

While the first problem is impossible to solve, the second one is fairly easy. So why leave those methods out? These methods are left out in order to make end-developers using EoD SQL think about what they are doing. ConnectedDataSet includes the indexOf and lastIndexOf methods, so if you really want to perform a remove(Object), you can use:

DataSet<Movie> movies = ...;
int index = movies.indexOf(movieToRemove);
if(index >= 0) movies.remove(index);

However I would strongly advise rather using an @Update method with a “DELETE” in it over this method (ie: get the database to do the work for you). You need to bare in mind that EoD SQL will never generate any SQL code for you, so it’ll never try creating a “DELETE” statement for you. The remove method is great for processing rows in an iterator, and then removing them (or some similar functionality), but be careful with it.

Conclusion

We’ve taken a deep, long look at the DataSet interface, and it’s implementation details. Swing applications that directly access a database almost always have the database locally, or within an intranet. This means that a ConnectedDataSet can be used to leverage the database resources, allowing the Swing application to seemingly deal with very large volumes of data very quickly. When dealing with smaller volumes of data in a Swing application, it may be an idea to use a normal Collection object; an array or a single object as a return type instead of a DataSet. In the next post we’ll take a look at another EoD SQL specific data-structure better suited to web-applications dealing with large volumes of data.

LinkedHashMap

Introduction

This is the first part of a series of posts, dedicated to how best to apply EoD SQL in different types of applications. Each post in this series will cover a specific type of application, and the different parts of EoD SQL that generally work best within those applications. The articles may dive fairly deep into the workings of the API, but at the end you’ll have a much better idea of where things fit in, and how they fit together.

EoD SQL well understands that (a) people like to do things differently and (b) one solution is not right for everything. While under the hood: any one part of EoD SQL works much the same way as any other part, as you climb the structural ladder, things begin to behave very differently. These behaviors have a massive performance and memory impact on your application and how you will be treating the underlying database drivers.

What we’ll be covering

1. Swing Applications
2. JSP / Servlets and “Related Technologies”
3. JavaScript / JSON Applications
4. GWT Applications

Although you could go with the “one size fits all” route (and EoD SQL would still perform wonderfully), the objective of these posts is to expose you to the different flavors of EoD SQL data structures and get you thinking about how you can mix and match them in your application to produce different results.

There are many articals out there detailing comparisons between C++ Templates and Java Generics (Google around and you’ll quickly find loads of them). Almost universally they view the erasure of Java generics as a fail-point. Even the ones written by “Java Gurus” seem to favour C++ templates as being more flexable. I recently stumbled upon a simple case that I’ve been doing in Java for ages, and while trying to implement the same concept in C++ discovered that it’s impossible.

The case is a generic Visitor model:

public interface Visitor<R, P> {
    R visitFirstNodeType(FirstNodeType node, P parameter);
    R visitSecondNodeType(SecondNodeType node, P parameter);
}

public abstract class AbstractNodeType {
    public abstract <R, P> R accept(Visitor<R, P> visitor, P parameter);
}

Looks simple enough, but try and implement it in C++ and you discover that you cannot use templates on a virtual method. At first this doesn’t make sense (even to a friend of mine who lives in C++). The simple reason is the way templates are handled: they are code generation shortcuts. Therefore the compiler needs to know the exact method to run, but by declaring the method virtual you are telling the compiler to deferre that descision until runtime. In Java the generics are “erased” (although they are kept in the signature), resolving to a single method implementation. This means that there is no problem trying to decide which method to run.

C++ can create the same structure without templates, but then you loose the type-safety contract that you defined in Java. It’s not really a fail-point on C++ (rather just a product of the implementation), but it is something that seems to be overlooked on the Java side of the fence. Generic erasure buys you fairly cool compile-time contracts.

EoD SQL doesn’t see very much development these days (although it sees plenty of downloads and usage). Mainly it’s very stable and does what it’s supposed to, so theres very little need to change things.

That said, there are a few issues that needed some attention, so theres the start of a new release (2.1). The new features and changes are as follows:

• The engine now caches some of the more expensive construction data with SoftReferences, dramatically reducing the cost of creating a Query implementation
• @Call methods may now return updatable DataSet objects by using the new “readOnly” annotation attribute
• @Select methods now have a useful “fetchSize” hint that will be passed down to the JDBC driver
• Much better error messages for invalid / unknown types and query parse errors
• Improvements and updates to all of the JavaDocs
• Some bug fixes and corrections

Like I said: theres not that much there, but the changes that do exist make life that little bit easier. If you have any ideas of features you’d like to see in this upcomming release, leave a comment or a feature request and we’ll have a chat about it (always on the lookout for good ideas)!