BigDecimal and your Money

I often see Java developers attempting to use the BigDecimal class to store monetary values for financial applications. This often seems like a great idea when you start out, but almost always comes back as a flesh-eating zombie intent on devouring your entire time some time later on. The BigDecimal class is designed around accuracy and has an almost infinite size. However: this choice will almost always come back to bite you, or someone else attempting to find a minor bug later on. It’s an especially bad idea when it comes to banking applications. So why is BigDecimal not well suited to storing monetary values then? For one thing: it doesn’t behave in a manner that is practical for financial systems. While this statement flies in the face of everything you’ve probably been taught about this class, it’s true. Read-on and I’ll tell you why.

Theres more numbers in there than you think!

A BigDecimal is not a standard floating point number. Instead: it’s a binary representation of a number. This means: 0.0 != 0.00. While this doesn’t seem like a problem at first: I’ve seen it cause no-end of strange little bugs. The only way to accurately determine whether two BigDecimal objects have an equal value is by using the compareTo method. Try these two little unit tests:

@Test
public void testScaleFactor() {
    final BigDecimal zero = new BigDecimal("0.0");
    final BigDecimal zerozero = new BigDecimal("0.00");

    assertEquals(zero, zerozero);
}

@Test
public void testScaleFactorCompare() {
    final BigDecimal zero = new BigDecimal("0.0");
    final BigDecimal zerozero = new BigDecimal("0.00");

    assertTrue(zero.compareTo(zerozero) == 0);
}

This technique works when you’re in control of the data and the comparison, but it breaks when you want to put a BigDecimal object into most other Java data-structures. I’ve actually seen someone use a BigDecimal as a key to a HashMap, which of course didn’t work. The solution in this case was simple: change the HashMap for a TreeMap and things were happy. However it won’t always be this simple.

They’re true high precision structures.

This doesn’t just mean that they are precise, it also means that they won’t run any calculation that wouldn’t result in a representable answer. Take the following code snippet as an example:

@Test
public void testArithmatic() {
    BigDecimal value = new BigDecimal(1);
    value = value.divide(new BigDecimal(3));
}

Primitive numeric types would just swallow this and represent the 0.3* as best they could, while a BigDecimal throws an ArithmeticException instead of attempting to represent a recurring number. In some cases getting an error will be desirable, but I’ve actually seen someone resolve the ArithmaticException like this:

try {
    return decimal1.divide(decimal2);
} catch(ArithmaticException ae) {
    return new BigDecimal(decimal1.doubleValue() / decimal2.doubleValue());
}

Yes folks, unfortunately I’m quite serious here. This is the sort of bug introduced by an error occurring, computations stop running, and someone adds a “hack” to just “make it work quickly and we’ll fix it later“. It’s a total disaster, but I see it far to often.

They don’t play nice with Databases.

According to the JDBC spec database drivers implement a getBigDecimal, setBigDecimal and updateBigDecimal functions. They seem like a great idea, until you ponder that your database may not have a suitable storage type for these values. When storing a BigDecimal in a database, it’s common to type the column as a DECIMAL or REAL SQL type. These are both standard floating-point types, with all the rounding errors that implies. They are also limited in capacity and will often overflow or cause a SQLException when attempting to store very large BigDecimal values.

The only practical solution which will keep all the BigDecimal functionality and accuracy in a database is to type the amounts a BLOB columns. Try to imagine the following table structure if you will:

CREATE TABLE transactions (
    initial_date DATETIME NOT NULL,
    effective_date DATETIME NOT NULL,
    description VARCHAR(30) NOT NULL,
    source_id BIGINT NOT NULL,
    destination_id BIGINT NOT NULL,
    in_amount BLOB NOT NULL,
    in_amount_currency CHAR(3) NOT NULL,
    effective_amount BLOB NOT NULL,
    effective_amount_currency CHAR(3) NOT NULL,
    charge_amount BLOB NOT NULL,
    tax_amount BLOB NOT NULL
);

That required four different BLOB columns, each one of which will be stored outside of table space. BLOB objects are very expensive both to store, and to work with. Each one often uses it’s own database resources (much like an internal cursor) to read or write the value. This translates to much more time and network usage between your application and it’s database. To add to the misery a BLOB is generally not readable by a SQL tool, one of the major reasons for sticking with a SQL database is that it can be managed from outside of your application.

Performance.

This is often raised as an issue, but ignored in favor of “accuracy”. The performance of BigDecimal is often considered “good enough” for general computing, and it’s fine if you want to add tax to an item every once in a while, but consider the number of interest calculations per month a moderate sized bank do. This may seem like an extreme case, but if your application ran a simple shipping and tax calculation for items on an online store in a JSP you’ve got effectively the same problem. In a very simple multiplication test BigDecimal performed over 2300 times slower than a simple long value. While this may only be milliseconds per mutation, a performance-factor of this size very quickly adds up to more computational time than is actually available to the system.

Also remember that BigDecimal (like most Number subclasses) are immutable. That means every calculation requires a copy of the existing BigDecimal. These copies are generally cleaned away by the eden-space collector (and G1 is very good at handling them), but when you put such a system into production it leads to a massive change in your heap requirements. Your BigDecimal objects must be allocated in such a way that a minimum number of them survive a garbage collection, the memory requirement of such a space quickly spirals out of control.

To add to the performance argument: the compareTo method is quite a bit slower than the equals method, and gets significantly slower as the size of the BigDecimal increases.

A Cure to BigDecimal Woes:

A standard long value can store the current value of the Unites States national debt (as cents, not dollars) 6477 times without any overflow. Whats more: it’s an integer type, not a floating point. This makes it easier and accurate to work with, and a guaranteed behavior. You’ll notice that several different behaviors in BigDecimal are either not well defined, or have multiple implementations. That said: depending on your application you may need to store the values as hundredths or even thousandths of cents. However this is highly dependent on your application, and theres almost always someone who can tell you exactly what unit the business works in. Bare in mind also that there are often de-facto (or even mandated) standards which exist between businesses about what unit of money they deal in, using more or less precision can lead to some serious problems when interfacing with suppliers or clients.

The mechanism I generally try to use is a custom-built MoneyAmount class (each application has different requirements) to store both the actual value, and it’s Currency. Building your own implementation opens the opportunity to use factory methods instead of a constructor. This will allow you to decide on the actual data-type at runtime, even during arithmetic operations. 99% of the time, an int or long value will suffice – when they don’t the implementation can change to using a BigInteger. The MoneyAmount class also enables you to define your own rounding schemes, and how you wish to handle recursive decimal places. I’ve seen systems that required several different rounding mechanisms depending on the context of the operation (currency pairs, country of operation and even time of day). For an example of this kind of factory discussion: take a look at the source-code for the java.util.EnumSet class. Two different implementations exist: the RegularEnumSet class uses a long to store a bit-set of all the selected constants. Given that very few enum values have more than 64 constants this implementation will cover most cases, just like a long will cover most requirements in a financial system.

Summary

This post is to warn people who are busy (or about to start) writing a system that will run financial calculations and are tempted to use BigDecimal. While it’s probably the most common type used for this purpose in the “enterprise” world, I’ve seen it backfire more times than I care to recount. My advise here is really to consider your options carefully. Taking shortcuts in implementation almost always leads to pain in the long-run (just look at the java.util.Properties class as an example of this).

Android User Interface Development

Android User Interface Development: A Beginners Guide is the title of the book that I’ve been writing. The main reason I haven’t recently had a chance to update this blog. The book is not just about the basics of good user interface design, but also how exactly these principals can be applied on an Android device. The book takes the concepts and in a practical manor applies them directly to various parts of the Android developers stack.

In broad terms, the book covers the following aspects:

  • Designing user-friendly interfaces that support quick and easy access to information
  • Exploring and implement multiple layouts in Android to design user interfaces for the different screen sizes and densities
  • Ensuring a consistent user-interface experience and improve your application performance by reusing your application components
  • Designing easy-on-the-eye themes for your Android applications
  • Displaying and select complex data structures from applications such as an address-book or calendar application by using Android widgets
  • Animating visual queues of what the application is currently doing, and what effect their actions are having
  • Customizing the built-in classes in Android to enhance the user interface by creating tabs and galleries
  • Learning to Leverage Android’s resource loading system
  • Learning how best to present your user with information; or capture information from them

You can find more information about the book, and also pre-order yourself a copy from the Packt Publishing web-site.

Unit Testing with Mock Objects and EoD SQL

Introduction

Unit Testing has become one of the corner-stones of good software development (it has been such for quite a long time actually). However writing a Unit Test that interacts with database code (at any level) can be very challenging. The main problem being: your data should be in a “known” state when the Unit Test starts running. In the past many approaches to this problem have been taken, such as: using an in-memory database; or DbUnit. Each approach has it’s up-sides and it’s down-sides (I won’t be going into them here).

To help your unit testing along EoD SQL is growing it’s own mocking extension (currently only in Subversion however). The EoD-Mock project allows you to build mock implementations of your database connection code without any external configuration. Thats right: no dependency injection; no JNDI; no factories; nothing!

How does it work?

If the EoD-Mock JAR file is on the class-path, EoD SQL will automatically look for mock implementations of your Query interfaces instead of generating a real implementation. For example, take a simple UserQuery interface:

public interface UserQuery extends BaseQuery {
    @Select("SELECT * FROM users WHERE email = ?1")
    User selectByEmail(String email);

    @Update(sql = "INSERT INTO users (email, username, birth_date) "
    + "VALUES(?{1.email}, ?{1.username}, ?{1.birthDate})",
    keys = GeneratedKeys.RETURNED_KEYS_FIRST_COLUMN)
    User insert(User user);

    @Select("SELECT * FROM users")
    DataSet<User> selectUsers();
}

To create a Mock of this interface using EoD-Mock all you need to do is write an implementation in the same package:

public class UserQueryMock extends AbstractMockQuery implements UserQuery {

    private final List<User> users = new ArrayList<User>();

    public UserQueryMock() {
        insert(new User("joe.bloggs@nowhere.com", "Joe Bloggs", new Date(83, 3, 6)));
        insert(new User("jeff@jeffswebsite.com", "Jeff Site", new Date(76, 8, 23)));
        insert(new User("logan@murkmurk.com", "Logan Sleep", new Date(90, 4, 1)));
    }

    public User selectByEmail(final String email) {
        for(final User user : users) {
            if(user.getEmail().equals(email)) {
                return user;
            }
        }

        return null;
    }

    public User insert(final User user) {
        final long id = users.size();

        final User clonedUser = new User(
                user.getEmail(),
                user.getUsername(),
                user.getBirthDate());

        clonedUser.setId(id);

        users.add(clonedUser);

        final User idUser = new User(null, null, null);
        idUser.setId(id);

        return idUser;
    }

    public DataSet<User> selectUsers() {
        return new MockDataSet<User>(users, false, true);
    }

}

Yup, it’s really as simple as that. Now if you ask EoD-SQL for an instance of UserQuery (ie: QueryTool.getQuery(UserQuery.class)): instead of generating an implementation, it will create an instance of UserQueryMock and return that.

How much does eod-mock take care of?

  • EoD-Mock will self-register as a QueryFactory with EoDSQL (but only under Java 6 and higher) if it’s on the classpath
  • A “default” DataSource is provided automatically. The provided implementation throws exceptions instead of providing database access
  • An AbstractMockQuery class is provided to take care of the methods declared in BaseQuery
  • A MockDataSet is provided, and will attempt to behave like a real DataSet object
  • QueryTool.getQuery will automatically return mocked query objects, meaning: no changes to your data-access layer

Current State

EoD-Mock is currently only available in the Subversion repository, but will be included in the next release of EoD SQL. There is no support currently for the DataIterator class, but that will no-doubt come very soon.

EoD SQL 2.1 Released

So I finally found some time to make a 2.1 release for EoD SQL. For those who don’t already know:

EoD SQL Allows for fast, simple binding between a Relation Database Query and Java objects.

Think of it as your friend that gets Hibernate or JPA out-of-your-face and lets you get on with actually writing some code.

The 2.1 release has a long list of changes behind it (which we’ve built over the last few months).

  1. It’s much faster than EoD SQL 2.0 was (which in turn was faster than 1.0).
  2. There are a good number of bug-fixes, making it nice and stable.
  3. The 2.1 release includes batch updates
    @Update(sql=”INSERT INTO users (name, email) VALUES (?{1.name}, ?{1.email})”,batchUpdate=true)
    void batchInsertUsers(Collection users) throws SQLException;
  4. GWT developers rejoice! EoD SQL can now fetch objects in any Collection type you choose!
    @Select(“SELECT * FROM users WHERE group = ?1”)
    ArrayList getUsersInGroup(long groupId);
  5. Byte arrays are now considered primitive types, and get mapped by EoD SQL out-of-the-box (no custom TypeMapper required)

That’s far from an exhaustive list, and you should really go download the API and try out some of the cool new features.

EoD SQL 2.1 – Now in First RC Release

EoD SQL just got a 2.1 Release Candidate kicked out the door. While there are (once again) many performance improvements and a few bug fixes, the big news of this release has got to be Batch Updates!

Some people (myself included) have been waiting a very long time to see this feature in EoD SQL, and it’s taken a fair amount of discussion to decide how it would work. The final answer came from Bernd Rinn: allow parameters to be specified as various Collection types. So to add a list of User objects to your database:

@Update(sql="INSERT INTO users (name, birth_date, password, email) VALUES (?{1.name}, ?{1.birth_date}, ?{1.password}, ?{1.email})",batchUpdate=true)
void insertUsers(List<User> users);

Batch updates will work with any class extending Collection, or an array type. It can also be used with simple parameters:

@Update(sql="REPLACE INTO user_group_join (user_id, group_id) VALUES (?{1.id}, ?{2.id})",batchUpdate=true)
void ensureUserInGroups(User user, Collection<Group> groups);

A Batch Update is generally much faster than a series of normal update invocations, because EoD SQL will use PreparedStatement.addBatch() to build up all of the parameters on the client-side before sending all of the data to the database in one statement. It doesn’t mean it’s a single transaction, but it does mean that the database can execute all of the parameter variations before sending any data back.

So why not go download the new release and give it a try.

Bernd Rinn

EoD SQL 2.1-Beta Released

EoD SQL 2.1-Beta has been released, you can go download it from the Documents & Files section of the homepage. The 2.1 release is about improving speed and usability over the 2.0 release. Some of the new features included are:

  • Specific Collection implementations (including those not in the standard Java API’s) can now be returned:
    @Select("SELECT * FROM users WHERE group = ?{1.id}")
     ArrayList<User> selectUserGroup(Group group);
  • Select methods can now populate existing objects:
    @Select(sql="SELECT * FROM user_meta_info WHERE user_id = ?{1.id}",into=1)
    void selectMetaInfo(User user);
  • Faster construction of query implementations
  • Better error reporting
  • Several small bug fixes

This is of course a beta release, so there may be bugs. That said the code is pretty well unit tested and should be relatively stable. One new feature not in the release (but on the way) is batch updates. Stay tuned for more information!

EoD SQL Applied – Part 5 / 5 (GWT Applications)

The Dreaded “Not Serializable” Problem

GWT turns all of your Java code into JavaScript, but it also obfuscates everything. With this in mind, it makes serialization a bit more complex than usual. GWT generates a serializer and deserializer for each class that could be transported across the wire to an RPC service. The difficulty comes in knowing which types had code generated, and which didn’t. GWT solves this problem for itself with the Serialization Policy file, where the compiler lists all of the classes that the client code will know how to deserialize.

This however leads to another problem: what happens when something unexpected gets in the way. Hibernate is the most used, and thus most complained about when it comes to Serialization problems. Hibernate turns any Collection object into a special “lazy” implementation that will only load your data when you ask for it. All very fine and well for a server bound application, but a GWT application needs all that data up front for serialization. When GWT comes to sending that List<Message> to the client, it chokes. It knows how to serialize an ArrayList, and LinkedList (since you worked with them on the client side), but a Hibernate List or PersistentCollection is a totally unknown type, so it’s not in the Serialization Policy, so the server side throws an Exception at you.

So how does EoD SQL help with these problems? Read on to find out! 😉 Read the rest of this entry »