Test: Overload Mulgara with TCP connections to test behavior of RMI

here are the next few related loglines:

2008-02-27 05:02:06,394 WARN  TransactionHelper(PLoSClinicalTrials)> rollback failed [TP-Processor136 org.topazproject.otm.util.TransactionHelper]
org.topazproject.otm.OtmException: error rolling back transaction
Caused by: java.net.ConnectException: Connection refused
2008-02-27 05:02:06,396 WARN  DBQueryFunction(PLoSClinicalTrials)> query '
                select obj.state from ObjectInfo obj where obj.id = :p1 ;
              ' execution failed. [TP-Processor136 org.plos.xacml.cond.DBQueryFunction]
org.plos.xacml.cond.DBQueryFunction$QueryException: query '
                select obj.state from ObjectInfo obj where obj.id = :p1 ;
              ' execution failed.
Caused by: org.topazproject.otm.OtmException: error starting transaction
Caused by: java.net.ConnectException: Connection refused
2008-02-27 05:02:14,946 ERROR ItqlStore(PLoSNTD)> Failed to close connection after rollback failure [TP-Processor73 been through net.sf.ehcache.constructs.w\
eb.filter.SimplePageCachingFilter org.topazproject.otm.stores.ItqlStore]
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode:
 faultString: java.net.ConnectException: Connection refused

it goes on in this manner for quite some time...

I also see these errors in the logs which may be related

2008-03-09 10:07:02,622 ERROR ItqlStore(PLoSONE)> Failed to close connection after rollback failure [TP-Processor207 org.topazproject.otm.stores.ItqlStore]
AxisFault
 faultCode: {http://schemas.xmlsoap.org/soap/envelope/}Server.userException
 faultSubcode:
 faultString: java.net.ConnectException: Connection timed out
 faultActor:
 faultNode:
 faultDetail:

Is there anything in the logs before which indicates something closed the connection for some reason?

Replying to amit:

Is there anything in the logs before which indicates something closed the connection for some reason?

We need to be careful with the term "connection" here, as it's being used for two somewhat different things. On the one hand there are the TCP connections, which is what is being referred to in the "java.net.ConnectException: Connection refused"; on the other hand there are "ItqlStoreConnection"s which is what is being referred to in the "Failed to close connection" message and which represent an OTM internal connection abstraction that has nothing to do with TCP connections, but instead designates a conversation spanning a transaction.

The "Failed to close connection" per se is not a huge deal - it means that the underlying mulgara Session object was not closed. This should not be particularly problematic in itself as the mulgara Session will get discarded when the http-session times out.

However, the "error rolling back transaction" just after the "Failed to close connection" message is indeed problematic: it means the transaction will stay active on mulgara (and hence block everybody else) until the transaction times out. So this needs to be investigated and fixed. Since the underlying cause is a "java.net.ConnectException" the problem is that mulgara is refusing connections, probably due to having run out of threads and the tcp backlog being full.

Replying to ronald:

However, the "error rolling back transaction" just after the "Failed to close connection" message is indeed problematic: it means the transaction will stay active on mulgara (and hence block everybody else) until the transaction times out. So this needs to be investigated and fixed.

it appears the fix is in the OTM transaction scope so assigning to Ronald.

Moving to 0.9 for diagnostics/testing with RMI. From Ronald:

Because 0.9 has a new mulgara and will be using RMI instead of SOAP, things are different for the two.

For the current stuff (0.8.*) I'm not sure there's much we can do, as I think it happens because mulgara is overloaded with tcp connections. The best I can think of is that we try and put such failed otm-connections on a queue and repeatedly try to roll-back the tx and close the session.

For 0.9 things are quite different because with the use of RMI we don't actually control tcp-connections (we strictly speaking don't now either, but the way axis is configured we effectively can). For 0.9 we need to run some test and figure out how things behave when we try to overload mulgara.

Assigning this back to Ronald to figure out if we can fix the multiple thread problem related to singleton parser.

Upgrading as it would be good to know load handling capabilities here.

With changes in mulgara from r1057 to r1069 all the problems related to concurrency and time-outs using MulgaraRMI.groovy have been fixed. MulgaraRMI can be used to overload Mulgara directly with lots of threads doing {delete/insert/query/[commit|rollback]} to simulate very heavy loads and to determine performance characteristics.