Odi's astoundingly incomplete notes

While researching about the scalability of the Wicket framework, I came across what seems like a widespread myth. In a couple of mailinglists and forums people were basically saying: "Scalability is basically limited by how much data is stored in a session. But memory is cheap, so don't worry". Sad, that this just slightly misses the point.

Scalability is limited by 4 things:

• CPU
• Memory
• Disk Throughput
• Network Bandwidth

Real applications typically hit only one of these limits. If we just look at the web framework, then the DB is out of our focus and we assume that this is not the bottleneck. If you're not doing video streaming then network bandwidth is not likely to be an issue for you. And if you're doing video streaming you better have plenty of spare bandwidth anyway. If you are not doing file serving (or again streaming) disk throughput will also not be an issue. What remains is CPU and memory. If CPU is your limit (the webapp does number crunching), then there is not much you can do: optimize your algorithms. So the only really interesting limiting factor is memory.

Now in Java we have a garbage collected heap. So memory is not completely independent from CPU. GC can consume a fair amount of CPU and it causes latency. So it is really important that the application does as little garbage collection as possible. In a generational GC heap (Concurrent Mark and Sweep - CMS) there is also cheap and fast garbage collection of the relatively small Eden space (the youngest generation) and expensive and slow garbage collection of the Old generation (the bulk of the heap). It's key that you prevent the big collections from happening. They really hurt. Rather have 10'000 small collections instead of 1 big one. That means that you should not "churn" objects, be conservative with what you allocate. And it means that the ones that you allocate temporarily (during a request or shorter) should have the shortest possible lifetime. Each object that survives the Eden collection is a performance killer: it has to be moved to the next generation and will have to be collected from there eventually by a big collection!

So if each request produces 1 MB of garbage and your Eden space is 100 MB, each 100 requests will cause a small collection. Fine if these are the only requests on the system. The requests are over, so all objects are unreferenced and can be collected. The collection will be quick and efficient.
If you now make 200 requests in parallel, the Eden space will be full half-way through the time. If those objects are still referenced now they can not be collected and will be moved to the next generation and you have hit the scalability limit: the Old generation will quickly fill up and big collections will have to run, comsuming CPU and causing latency. If however most of these objects are no longer referenced, they can be collected quickly and your webapp still scales.

What about session state now? Session state doesn't change much usually. So it is more or less constant. Its objects will live in the Old (or even permanent) generation. It's certainly a bad idea to constantly add and remove objects to/from the session, because these objects will have to be collected in the Old generation by a big collection. Session state is also really small usually. Just a few KB. Do the maths: a 2 GB heap can hold 20'000 sessions of 100 KB. That's more than enough and will not be the bottleneck. Of course don't store huge data in sessions.

To sum up this posting: if you want a scalable Java webapp:

• reduce object allocation
• no object churning
• prefer short-lived objects
• clear references as soon as possible
• keep response times as short as possible: longer response time means that request scoped objects may survive a small GC
• keep session state small
• rarely add/remove objects to/from session
• use the CMS garbage collector
• size the eden, old and permanent generations sensibly (100MB, 2GB, 64MB is a good example) and inspect their use with JConsole

For frameworks that means:

• careful with state save/restore: this creates a lot of garbage for every request
• careful with parsers: they can create a lot of garbage
• careful with HTML formatting: string ops easily create garbage
• careful how much you keep during the lifecycle of a request (request, response, context objects)
• careful with filter/interceptor architectures (streams): they tend to copy data with every added layer, plus add to the stack
• careful with (stream) buffering: buffers are relativly big chunky objects likely to end up in the Old generation or exhaust Eden space quickly.

Java objects that are infamous for a lot of overhead in terms of object allocation:

• Iterator (Java 5 shorthand for loops!): when used heavily e.g. to recursively iterate over tree structures
• HashSet, HashMap, Hashtable, LinkedList: wrap each inserted object
• ByteArrayOutputStream.toByteArray(): copies the internal array
• ArrayList: allocates new array when growing
• XML parsers and transformers (XSLT)
• String operations
• some JDBC drivers like Oracle
• reporting libraries like JasperReports

posted on 2009-03-27 01:03 UTC in Code | 1 comments | permalink

Today Gentoo released some updates that cause a conflict. It's resolution may damage your system:

[ebuild     U ] sys-fs/e2fsprogs-1.41.2 [1.40.9] 
[ebuild  N    ] sys-libs/e2fsprogs-libs-1.41.2  USE="nls" 
[ebuild     U ] net-fs/nfs-utils-1.1.3 [1.1.0-r1] 
[blocks B     ] sys-libs/ss (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] <sys-fs/e2fsprogs-1.41 (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] sys-libs/com_err (is blocking sys-libs/e2fsprogs-libs-1.41.2)
[blocks B     ] sys-libs/e2fsprogs-libs (is blocking sys-libs/ss-1.40.9, sys-libs/com_err-1.40.9)

The reason for the block is that ss and com_err have both been integrated into e2fsprogs-libs. Blocks are normally not a problem and happen from time to time, and can easily be resolved. Also this one can seemingly easily be resolved by unmerging ss and com_err and then merging e2fsprogs-libs. But ss and com_err are part of the system profile and unmerging them even for a short time is dangerous.
Be really careful NOT to unmerge sys-libs/ss and sys-libs/com_err from your system! These are critical core libraries that are used by dozens of programs (such as wget, curl, openssh,  apache, postfix), which will be broken after unmerging. There is a manual upgrade path in comment#7 of bug 234907. But I suggest you wait until this is resolved automatically by the next version of portage. Pretty sure the Gentoo developers will fix this in the portage tree soon.

Well, they haven't so far. So here is the paranoid way to do it:

# don't depend on the net later
emerge -fuD world
# backup in case something goes wrong
quickpkg ss com_err
# unmerge
emerge -C ss com_err e2fsprogs
# merge
emerge -1 e2fsprogs-libs
emerge e2fsprogs
# fix broken things
revdep-rebuild

posted on 2008-10-28 10:34 UTC in Code | 37 comments | permalink

Something to remember when killing an Oracle instance manually (which should not be done really): clean up the shared memory allocations. Use the command ipcs to list the used shared memory segments, and ipcrm to remove them. Alternatively reboot the system.

If you fail to do that, sqlplus would not startup anymore and just consume a lot of CPU, instead of telling you that it can not allocate shared memory.

posted on 2008-10-17 10:28 UTC in Code | 1 comments | permalink

 Unlike older releases, Oracle 11g sets password expiry by default. That's really annyoing

• for SYS and SYSTEM: noboday wants to regularly change (and forget) them.
• for any accounts that are used by application servers: the password is usually configured in a datasource file, nobody would ever want to change it
• if some superusers directly access the DB and keep forgetting the password, the automatic locking will lock your application out: not great for your uptime.

So let's get rid of these annoyances with:

ALTER PROFILE DEFAULT LIMIT
  FAILED_LOGIN_ATTEMPTS UNLIMITED
  PASSWORD_LIFE_TIME UNLIMITED;

And also let's turn off the default auditing:

NOAUDIT ALL;
DELETE FROM SYS.AUD$;

posted on 2008-10-03 18:26 UTC in Code | 14 comments | permalink

The whole world, please stop generating WSDLs! It's bollocks.

A WSDL file is an interoperability document. Other applications depend on it. For this reason its content must be fully under your control. The exact content of this file must not depend on some half-baked decision of some stupid WSDL generator that doesn't know what it's doing.

The real world out there is unfortunately as full of bad WSDL code generators as it is full of broken WSDL interpreters. Some do not properly support different namespaces. Some only accept a single embedded schema. Of course most only support SOAP binding. Some expect certain namespace prefixes etc. The bugs and madnesses are endless. So you must be able to tweak this file to work around certain interoperability issues. Sometimes even the users of your service may have to pre-process the WSDL with an XSLT transformation to accomodate for certain bugs. Use your favourite WSDL edtior. But be prepared to make manual changes in the XML file if necessary.

At the same time, be careful to be pragmatic about which WS features you use. The fancier stuff you do the more problems you will have to face.


posted on 2008-06-16 17:47 UTC in Code | 0 comments | permalink

This is a funny story. We experienced a strange Oracle error last night:
ORA-24816: Expanded non LONG bind data supplied after actual LONG or LOB column
I had never met that beast before. Ignore the fact that the error message is completely cryptic. Ignore the fact that Oracle would never mention the name of the table and column that causes the problem. The table in question was easy enough to find. It contains about 70 columns (a typical number for business data from a legacy system). Three of them had recently been converted to CLOBs, as they could exceed 4000 characters.

At a first glance the error looks completely random and really looks like a driver issue. But the case was at least reproducible with some well-known data. So I considered an Oracle JDBC driver update to 10.0.2.0.4. Looking at the long changelog that promises fixes for some scary bugs I hoped that they had fixed this one as well. But no, the error still occurs with this latest version.

As some postings suggested, rearranging the column order (can you believe it!!!) might help. Well, we are not controlling the SQL manually, but EJB3 with Hibernate as an implementation does that in the backend. Turning on SQL logging revealed that Hibernate chose to arrange the columns so that two LOB columns were next to each other. Interestingly the two columns were also adjacent to each other in the entity bean definition (mappping). In a desperate act I just rearranged the getters and setters for this field, so that there were plenty of non-LOB columns in between the two LOBs. It seems to work.

UPDATE: Actually it turns out that Oracle says it's a limitation, and that LONG bind variables must come last in a statement. That's easily done by rearranging the getters and setters in Hibernate. Unfortunately there is one case where that doesn't work: If you use joined inheritance, Hibernate will put the primary key join column last. And there is no way to change that, other than patching Hibernate.

If you can learn anything from that, then it is how crucial of a feature it is for an O/R mapping framework to give the developer complete control over the generated SQL if he needs to. It's the number one mistake of O/R frameworks to not allow that. This poses a very hard wall that you will sooner or later hit in any project.


posted on 2008-06-05 15:04 UTC in Code | 2 comments | permalink

I have some JUnit test that exercise and compare the performance of certain classes. They usually look like this:

long start = System.currentTimeMillis();
test1();
long end = System.currentTimeMillis();
long duration1 = end - start;
start = System.currentTimeMillis();
test2();
end = System.currentTimeMillis();
long duration2 = end - start;
assertTrue(duration1 > duration2);

This is useful to detect performance regressions. During development of these tests I came across a very obvious but interesting fact: do not use differences of the system timers for these tests. With system timers I mean System.currentTimeMillis() and System.nanoTime().

Rather the ThreadMXBean.getCurrentThreadCpuTime() method should be used. This makes the test independent of the load on the system, most of scheduling artifacts and even the garbage collection. All in all I get very stable and consistent result from this accounting timer. So my tests now look like this:

ThreadMXBean mx = ManagementFactory.getThreadMXBean();
long start = mx.getCurrentThreadCpuTime();
test1();
long end = mx.getCurrentThreadCpuTime();
long duration1 = end - start;
start = mx.getCurrentThreadCpuTime();
test2();
end = mx.getCurrentThreadCpuTime();
long duration2 = end - start;
assertTrue(duration1 > duration2);

posted on 2008-06-02 15:47 UTC in Code | 0 comments | permalink

Here is a simple init script for Oracle 10/11 on RedHat Linux. It assumes your instances to start/stop are listed in /etc/oratab and the oracle user's shell environment sets ORACLE_HOME correctly.

#!/bin/sh
#
# chkconfig: 345 98 10
# description: Oracle
#

#
# change the value of ORACLE to the login name of the
# oracle owner at your site
#

ORACLE=oracle

case $1 in
'start')
        cat <<-"EOF"|su - ${ORACLE}
        # Start Oracle Net
        if [ -f ${ORACLE_HOME}/bin/tnslsnr ] ;
        then
             echo "starting Oracle Net Listener"
             ${ORACLE_HOME}/bin/lsnrctl start
        fi
        echo "Starting Oracle databases"
        ${ORACLE_HOME}/bin/dbstart
        ${ORACLE_HOME}/bin/emctl start dbconsole
EOF
        ;;
'stop')
        cat <<-"EOF"|su - ${ORACLE}
        echo "shutting down"
        ${ORACLE_HOME}/bin/emctl stop dbconsole
        # Stop Oracle Net
        if [ -f ${ORACLE_HOME}/bin/tnslsnr ] ;
        then
             echo "stopping Oracle Net Listener"
             ${ORACLE_HOME}/bin/lsnrctl stop
        fi
        echo "stopping Oracle databases"
        ${ORACLE_HOME}/bin/dbshut
EOF
        ;;
*)
        echo "usage: $0 {start|stop}"
        exit
        ;;
esac
#
exit

posted on 2008-05-08 14:20 UTC in Code | 0 comments | permalink

I don't know what causes this, but have a look at the latest Linux kernel performance benchmark results yourself. A 30% (!) performance gain of a Java benchmark is impressive. My guess is the new scheduler is responsible for it. Could make a huge difference for application servers that typically run hundreds of threads. It looks like it could have to do with VMs using sched_yield() and a kernel parameter controlling its behaviour.

posted on 2008-01-06 14:26 UTC in Code | 2 comments | permalink

Roughly 24 days. That's the number of milliseconds that can be represented by a Java integer. Good to know when you have to be careful of int overflows in millisecond arithmetic and when you can allow for sloppyness.

posted on 2007-12-19 10:41 UTC in Code | 0 comments | permalink