After a brief discussion with Rich he agreed to enable JMX on the various
servers (JMX is basically SNMP on steroids for Java). This will allow us
to take a peek inside the JVM's at any time, such as to determine heap usage,
cache usages, look at the tomcat connectors, etc., and even to do things
like clear caches (ehcache has JMX support as of 1.3, which we haven't
upgraded to; oscache supports JMX from version 2.4 onwards, which we haven't
upgraded to either yet).
The overhead of JMX when not being queried is basically not measurable;
the overhead when a client is attached depends on how hard the client
hits the server, but something like jconsole (which extracts a lot of
information and therefore has a fairly large impact) is around 3% - 5%.
There are various JMX clients out there; jconsole is part of the jdk;
jManage is a web-based client;
JMX Browser
supports both web-based clients and normal GUI; JMX support is available
in some general monitoring packages such as nagios.
JMX is enabled by setting a few system properties. JMX can be enabled
for "direct" contact only (requires privileges to attach to the JVM
process) or for network access - the latter is preferred since it
would allow us developers to take a look at things too. Additionally,
when opening a tcp port a couple security options can be configured:
SSL and authentication. These require some more setup, though. I'm
going to describe the easy setup (no ssl or authentication) here,
assuming you trust your firewall enough.
The config needed to enable JMX are the following three defines in
the startup options /etc/sysconfig/<server>:
-Dcom.sun.management.jmxremote.port=21212
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
(pick any port you like, of course).
Specifically, for mulgara put the following in /etc/sysconfig/mulgara:
export mulgara_TOMCAT_OPTS="-Xmx400m -XX:MaxPermSize=256m -Dcom.sun.management.jmxremote.port=21212 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
For topaz put the following in /etc/sysconfig/topaz:
export topaz_TOMCAT_OPTS="-Xmx2000m -Dcom.sun.management.jmxremote.port=21212 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
For plosone put the following in /etc/sysconfig/plosone:
export plosone_TOMCAT_OPTS="-Xmx3818m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote.port=21212 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false"
For fedora there's currently no way to add these defines except by
editing /usr/local/topaz/fedora/server/bin/fedora.sh directly.
