Which Java VM scales best?

I find it ironic that the one company with a not-so-secret objective to “kill cross-platform Java” (according to an internal Microsoft document; see Resources below) also is the Java vendor that kept our 100% Pure Java company (Volano) in business for the past year and a half. Without the impressive performance of Microsoft’s Java virtual machine, we would not have acquired the many VolanoChat customers who run our chat server on high-traffic Web sites. My article in the December issue of JavaWorld summarized our experience and reported the performance scores of four Java virtual machines using our Java server benchmark, VolanoMark 1.0.

While writing that article in November, I suggested to a Microsoft product manager that Sun might eventually trade places with Microsoft as the Java performance leader. “It’ll never happen,” was his quick reply. Although it might still be hard to argue with his response, most of the Java VM vendors have significantly narrowed Microsoft’s lead.

Mostly in response to the December article, I have received more than 450 e-mail messages from 19 Java vendors trying to solve the technical problems they encountered while running VolanoMark and various Java server applications that the benchmark represents. Their efforts are now starting to reduce the great disparity that exists between Java virtual machines commonly used on the Internet today.

What is VolanoMark?

VolanoMark is a 100% Pure Java server benchmark characterized by long-lasting network connections and high thread counts. In this context, long-lasting means that the connections last several minutes or longer, rather than just a few seconds. The VolanoMark benchmark creates client connections in groups of 20 and measures how long it takes for the clients to take turns broadcasting their messages to the group. At the end of the test, it reports a score as the average number of messages transferred by the server per second. Its results have accurately predicted the real-world Java performance and scalability limits of our VolanoChat product line for almost two years now.

VolanoMark 2.0, which adds scalability measurements, is an update to the tool and is the first benchmark to be submitted to the SPEC Open System Group for use in its server-side Java benchmark suite. There are other Java benchmark programs under consideration for the SPEC suite that have different characteristics, such as the high connection-turnover rates of a Java Web server or the transaction-processing requirements of a Java database.

When trying to determine whether a product meets your performance and scalability requirements, the best approach is to write your own benchmark and obtain your own results. The next best approach is to run a benchmark program yourself on your own system. VolanoMark 1.0, for example, is available as a free download from Volano’s Web site. (See the Resources section below.) In any case, when you read benchmark reports you should make sure you understand the characteristics of the test and the configuration of the system. Without information about the hardware configuration, the operating-system settings, and the Java virtual machine heap and stack options, benchmark scores are meaningless. Fortunately, SPEC will provide a set of well-defined rules for running its server-side Java benchmark suite when it becomes available early next year.

The benchmark tests

The tests presented in this report look at the scalability of 12 Java virtual machines on eight operating systems, using a common Intel hardware platform when possible.

The connection scalability tests are divided into two groups of Java virtual machines (VMs):

• Java VMs available now as final production releases
• Java VMs scheduled for release later this year

The processor scalability tests compare Microsoft’s virtual machine on Windows NT with Sun’s virtual machine on Solaris Intel Edition and Solaris SPARC Edition. Note that these processor scalability tests ran on different hardware than all the other tests.

The final tests show a direct comparison of all the Java virtual machines, using the local loopback interface with the client and server application running on the same machine.

Except for the processor scalability results, all tests ran the VolanoMark server on a 200-megahertz (MHz) Intel Pentium Pro processor with a 256-kilobyte (KB) L2 cache and 256 megabytes (MB) of RAM. The client connected to the server machine via a 10-megabit-per-second (Mbps) Ethernet network. For the VolanoMark network test, the client side ran on a 200-MHz Intel Pentium Pro processor with a 256-KB L2 cache and 128 MB of RAM using Microsoft’s SDK for Java version 3.0 Pre-Release 1 (jview version 5.00.2750). The Apple JVM tests ran on a Apple Power Mac G3.

We ran the VolanoMark server and client applications with an increased initial heap size of 8 MB, an increased maximum heap size of 64 MB, and a decreased native stack size of 32 KB (using java -ms8m -mx64m -ss32k), except as follows:

• The Microsoft SDK has no heap and stack command options available.
• The Apple MRJ failed to run with the increased heap sizes.
• The Novell JDK failed to run with a stack size less than its 48-KB default.
• We did not decrease the 4-gigabyte (GB) default maximum heap size of the TowerJ-generated executable.

With the exceptions stated above, the commands were entered as follows, starting with two connections and working up to 900 connections:

Server

java -ms8m -mx64m -ss32k COM.volano.Main

Client

jview COM.volano.Mark -rooms 1 -users 2 -count 10000
jview COM.volano.Mark -rooms 5
  ...
jview COM.volano.Mark -rooms 45

See the COM.volano.Mark command synopsis for a complete description of all options.

To give you an idea of the heap usage, consider that the VolanoMark server uses 6,136 KB of heap memory when it runs under JavaSoft JDK 1.1.6 with 200 connections (-rooms 10) and with the garbage collector disabled.

All connection scalability tests were performed without restarting the server between tests, except for the JavaSoft JDK 1.1.6 and Apple MRJ 2.0 JVMs, which required a restart in order to reach 900 connections. Restarting these JVMs may have caused a variation in their scores due to the effects of the garbage collector, but not enough to alter the general picture of the results.

In this article, the SunSoft JDK refers to what Sun calls the Production Release for Solaris. The JavaSoft JDK refers to Sun’s JDK 1.1 Win32 Release. See the Resources section for links to the download locations of each JVM.

Current Java VMs: Free and available now

These seven Java virtual machines are the ones we recommend to our customers for running the VolanoChat product live on production Web sites. The chart shows the VolanoMark 2.0 scores for seven Java VMs as the number of connections are increased. The number of connections are shown across the bottom, with the throughput in messages per second shown on the left. The scores are shown at up to 900 concurrent connections. Failed tests in the table are indicated by “---” (no score shown).

Microsoft SDK 2.02 still stands alone as the only fast and scalable Java virtual machine. Our customers with the highest Web site traffic currently have no other viable choice for a JVM. I had to restart the JavaSoft JDK 1.1.6 virtual machine five times between tests in order to reach 900 concurrent connections, and none of our large customers have succeeded in running our VolanoChat product with JavaSoft’s JVM on Windows NT. IBM’s recent updates to its operating system, TCP/IP stack, and JVM have given it a huge 50 percent performance improvement over the VolanoMark 2.0 tests I ran without the updates (not shown here), but I could not get more than 400 concurrent connections with IBM’s JVM.

We had been waiting for more than a year for a stable Solaris Java VM and Sun finally delivered.

The SunSoft JDK 1.1.5 VM for Solaris may be a bit laggard in its performance, but its reliability is rock solid. We had been waiting for more than a year for a stable Solaris Java virtual machine with native threads and a just-in-time compiler, and Sun finally delivered on April 6, 1998. For me this was the biggest Java news of this year, but it went completely unnoticed (perhaps because its robust performance was not immediately apparent to the trade press that covered its release). After working with Java virtual machines on the server since early 1996, I had begun to lower my expectations of quality — but the SunSoft JDK 1.1.5 sets a new standard for stability. No other JVM comes close. We ran our VolanoChat demonstration server on this JVM for more than 32 days non-stop — that’s roughly 40 messages per second, 24 hours per day for 32 days, without any down time and without even going over its initial 8-MB heap size! I think it would have run forever, but I had to bring it down in order to upgrade the VolanoChat server. (In fact, we haven’t rebooted the Solaris 2.6 operating system since we first set it up on our Internet Web and chat server machine on November 5, 1997.)

Apple’s Java virtual machine performed better than I had expected, although I had to restart the server between each test. The Linux and FreeBSD virtual machines do quite well considering that neither of them have a just-in-time compiler or native threads. For both Linux and FreeBSD, I was unable to get 300 concurrent connections or more, even after I increased their per-process file descriptor limits to more than 1000. In addition, the Linux and FreeBSD VMs have problems with the new Java 1.1 socket timeout feature, although I have been unable to isolate the cause. With the socket timeout feature enabled in the VolanoMark test, the Linux JVM runs extremely slowly and the FreeBSD JVM crashes with a core dump. Disabling the feature allowed them both to run the tests successfully.

Microsoft SDK 2.02

• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
• Microsoft Internet Explorer 4.01 SP1 (Version 4.72.3110.8)
• Microsoft SDK for Java Version 2.02
• Microsoft virtual machine for Java (jview version 4.79.2405)
• Ran clspack -auto after installation (WINNTJavaClassesclasses.zip is 7,702,544 bytes)
• Set foreground application performance boost = None.
• No heap or stack command options are available.

JavaSoft JDK 1.1.6

• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
• JavaSoft JDK Version “JDK1.1.6N”
• JIT Update for JDK 1.1.6 Early Access 2 (Symantec JIT Version x3.00.050)
• Set foreground application performance boost = None.

IBM JDK 1.1.6

• IBM OS/2 Warp 4.00 FixPak 7 (Revision 9.031)
• IBM TCP/IP Version 4.1 (SOCKETS.SYS: 5.3003, AFOS2.SYS: 5.3000, AFINET.SYS: 5.3002)
• IBM JDK Version “JDK 1.1.6 IBM build o116-19980605 (JIT: javax)”
• Set THREADS=4095 in CONFIG.SYS.

SunSoft JDK 1.1.5

• Sun Solaris 2.6 Desktop Intel Platform Edition with Maintenance Update 1
• Solaris patch number 105491-04, “Dynamic linker patch (Intel)”
• SunSoft JDK Version “Solaris_JDK_1.1.5_02”
• Increased file descriptor limit to 4096.

Apple MRJ 2.0

• Apple Power Macintosh G3 with 266-MHz PowerPC and 96 MB of RAM
• Apple Mac OS 8.1
• Apple Mac OS Runtime for Java 2.0 (with default heap and stack settings)

Linux JDK 1.1.6

• Red Hat Linux 4.2 (Linux Kernel 2.0.30)
• Linux JDK Version “Linux_JDK_1.1.6_v2”
• Increased file descriptor limit to 1024.
• Had to disable the use of the Java 1.1 socket timeout feature by setting the VolanoMark property server.timeout=0.

FreeBSD JDK 1.1.5

• FreeBSD Version 2.2.6-RELEASE
• FreeBSD JDK Version “jdk1.1.5-Freebsd:02/25/98”
• Increased file descriptor limit to 4096.
• Had to disable the use of the Java 1.1 socket timeout feature by setting the VolanoMark property server.timeout=0.

Looking ahead: Java VMs arriving later this year

These Java virtual machines should be available at various times throughout the year. Check with the Java vendors themselves for specific availability dates. Keep in mind that none of these JVMs are yet in their final release, so performance characteristics are likely to change.

One quick look at the chart and you’ll see that the results are starting to get boring. That’s a good thing. It means that Java developers can start to expect the same kind of performance, scalability, and reliability in Java virtual machines regardless of the platform or Java vendor. It used to be the joke that Java was “write once, run only on Solaris and NT.” That’s no longer true, thanks to Tower Technology and Novell.

I was surprised

to see Tower and

Novell come out

of nowhere with

such great VolanoMark

performance and

scalability.

It has taken anywhere from four months to as long as a year from the time Java vendors first run VolanoMark until the time they run it well, so I was surprised to see Tower and Novell come out of nowhere with such great VolanoMark performance and scalability. To be fair, they had the advantage of coming to market when issues such as server-side Java performance had already become important.

Tower is working to determine the cause of its failure at 400 concurrent connections on Linux, but it is likely a limitation in the operating system itself. TowerJ runs on several other platforms with no such limitation. Apart from this, all of the Java virtual machines ran without any errors at all. In fact, I found the JavaSoft JDK 1.2 Beta 3 virtual machine to be more stable than the currently released JavaSoft JDK 1.1.6.

The limitation in Windows NT 4.0 of 2048 threads per process causes the VolanoMark server to get a java.lang.OutOfMemoryError when trying to get 1000 concurrent connections on Windows NT. The Solaris JVM uses a different threading model, and I have run the test myself with no problems up to 2000 concurrent connections. Sun told me that they have run the test with as many as 2400 concurrent connections, and they are working to go beyond that limit as well. Novell’s JVM seems to be limited only by the amount of real memory on the system, although I have not tested to see what the actual limit might be with 256 MB of RAM.

Tower TowerJ 2.1.2

• Red Hat Linux 4.2 (Linux Kernel 2.0.30)
• TowerJ Version 2.1.2.0 x86-linux
• Used command option -b-heap-min 8388608 to set initial heap size to 8 MB.

Novell JDK 1.1.5

• Novell NetWare Prototype 5.00 Beta 3.0 (Build 1158, 19 June 1998)
• Novell Java Version 1.1.5a (18 June 1998)
• Symantec Java! JustInTime Compiler Version 3.00.040(x) for JDK 1.1.x
• Set Maximum Packet Receive Buffers = 1000 (default is 500).
• Used default native stack size of 48 KB since the test failed with 32 KB.
• Set the -as NetWare JDK command option. According to Novell, the -as platform-specific command option provides an important load-balancing function that will be enabled by default in the final version of the Novell Java VM for NetWare 5.

Microsoft SDK 3.0 P1

• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
• Microsoft Internet Explorer 4.01 SP1 (Version 4.72.3110.8)
• Microsoft SDK for Java Version 3.0 Pre-Release 1
• Microsoft virtual machine for Java (jview version 5.00.2750)
• Ran clspack -auto after installation (WINNTJavaClassesclasses.zip is 7,012,092 bytes)
• Set foreground application performance boost = None.
• No heap or stack command options are available.

JavaSoft JDK 1.2 Beta 3

• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
• JavaSoft JDK Version “JDK-1.2beta3-N”
• Symantec Java! JustInTime Compiler Version 3.00.023(x) for JDK 1.2
• Set foreground application performance boost = None.
• Used -Xms8m -Xmx64m -Xss32k for non-standard command options.
• Set -Djava.compiler=symcjit to enable the Symantec just-in-time compiler.

SunSoft JDK 1.2 Dev 3

• Sun Solaris 2.6 Desktop Intel Platform Edition with Maintenance Update 1
• Solaris patch number 105491-04, “Dynamic linker patch (Intel)”
• SunSoft JDK Version “Solaris_JDK_1.2_01_dev03”
• Used -Xms8m -Xmx64m -Xss32k for non-standard command options.

The hardware solution

If you tried to buy better performance last year by adding processors, you might have been surprised by the results! Java performance early last year was strictly a software problem, and additional processors just made the problem much worse.

As you can see, those problems are fixed. The SunSoft JVM for Solaris scales well on both the Intel and SPARC platforms. The Microsoft JVM scales less well, especially on the fourth processor. Again, the virtual machines tested here are not in their final release, so these processor scalability results might change.

The highest VolanoMark result I have seen was a score of 5,642 messages per second on a Sun Ultra-Enterprise 4000/5000 with four 336-MHz UltraSPARC processors and 4 GB of RAM. I was afraid to find out what that machine might cost, though.

Microsoft SDK 3.0 P1 Intel

• Dell PowerEdge 6100/200
• 4 x 200 MHz Intel Pentium Pro with 512 KB of L2 cache and 128 MB of RAM
• Microsoft Windows NT Workstation Version 4.0 (Build 1381: Service Pack 3)
• Microsoft SDK for Java Version 3.0 Pre-Release 1
• Microsoft virtual machine for Java (jview version 5.00.2750)
• Set foreground application performance boost = None.

SunSoft JDK 1.2 Dev 3 Intel

• Dell PowerEdge 6100/200
• 4 x 200-MHz Intel Pentium Pro with 512 KB of L2 cache and 128 MB of RAM
• Sun Solaris 2.6 Server Intel Platform Edition
• SunSoft JDK Version “Solaris_JDK_1.2_01_dev03”

SunSoft JDK 1.2 Dev 3 SPARC

• Sun Ultra Enterprise 450
• 4 x 248-MHz UltraSPARC-II with 512 MB of RAM
• Sun Solaris 2.6 Server SPARC Platform Edition
• SunSoft JDK Version “Solaris_JDK_1.2_01_dev03”

The Race

I wanted to emphasize the scalability and not the raw performance of the Java virtual machines in this follow-up article, but I couldn’t help but line them all up and compare them side-by-side again. All Java virtual machines shown below ran the same VolanoMark local loopback test on the same 200-MHz Intel Pentium Pro machine with 256 MB of RAM — except for the Apple MRJ 2.0, which ran on a Power Mac G3. (For complete details of each test environment, see the lists above.) Apart from the Apple JVM, this is a direct, apples-to-apples comparison, with each vendor’s JVM and operating system sandwiched between identical hardware and Java application code.

Where possible, the actual commands used to run the VolanoMark server and its client test driver were:

Server

java -ms8m -mx64m -ss32k COM.volano.Main

Client

java -ms8m -mx64m -ss32k COM.volano.Mark -count 100

except for the Microsoft SDK, Apple MRJ, and Novell JDK, as noted above in the description of the benchmark tests.

To give you an idea of the heap usage, the server uses 26,432 KB of heap memory in this test (40 percent of the 64-MB limit) when the garbage collector is disabled using JavaSoft JDK 1.1.6.

Tower shows that, for pure speed on the server side, a Java-to-native compiler is the way to go. The TowerJ compiler generates a native executable from pure Java code that runs almost 25 percent faster than the fastest Java interpreter, even when the interpreter has a just-in-time compiler. However — unlike all the other listed vendors’ offerings — Tower’s technology is not free, with prices ranging from 9 to ,999.

The rest of the virtual machines cover a full order of magnitude in the range of performance scores.

Microsoft’s bold “it’ll never happen” claim still holds true, and Microsoft comes out on top again among all the Java virtual machine interpreters. In a surprise, Novell comes in right below Microsoft with a beta release of its NetWare 5 Java virtual machine. JavaSoft’s JDK 1.2 should be an excellent alternative for those unwilling or unable to use Microsoft’s JVM because of its lack of Java Native Method Interface (JNI) and Remote Method Invocation (RMI) capabilities. And IBM has delivered excellent performance in its updated JDK 1.1.6 for OS/2 Warp.

It took me just one afternoon to install and verify VolanoChat on an operating system I had never even seen before!

Conclusion

If you’ve been reading lately about the death of Java’s “write once, run anywhere” promise, don’t you believe it. It took me just one afternoon to install and verify our VolanoChat product on an operating system I had never even seen before! Thanks to Novell’s good work, we get another 4 million potential customers for free. And our applet runs on a wide variety of platforms — a feat that was inconceivable just three years ago, before Java was invented.

We have customers doing just fine running VolanoChat servers on all of the currently released Java virtual machines discussed here. But with the current batch of VMs, customers are forced to make a choice between rock-solid stability (Sun) and blinding speed (Microsoft). According to my tests, though (and the latest VM release schedules), we will have at least five Java virtual machines by December that deliver the speed, stability, and scalability that the Java platform deserves.