JSON Objects Used

• Standard items used in all all of the JSON Objects: (including annotation creation)
  • actionErrors
  • numFieldErrors
  • fieldErrors
• Display of annotations
  • annotationId
  • annotation.commentTitle
  • creatorUserName
  • annotation.creator
  • annotation.createdAsDate
  • annotation.escapedTruncatedComment
• Ratings
  • insight
  • reliability
  • style
  • commentTitle
  • comment
• New Discussion
  • annotationId

Parameters Submitted

• Standard item that gets submitted:
  • articleURI
• Create Annotation
  • target
  • startPath
  • startOffset
  • endPath
  • endOffset
  • commentTitle
  • comment
  • isPublic
• Rating
  • commentTitle
  • comment
  • insight
  • reliability
  • style
• Flagging
  • target
  • comment
  • reasonCode
  • response
• New Discussion
  • target
  • commentTitle
  • comment
  • isPublic

• Discussion Replies
  • root
  • inReplyTo
  • commentTitle
  • comment

Annotation Creation and Resolution

The basic theory behind the annotations is that we somehow map a location in the (X)HTML document to a location in the XML document. To do this, we use xpointers. On the client side, for every major element, we put the corresponding xpath location of the XML element. For instance, the second paragraph in the first section of the article will have an attribute on the <p> in the html that gives the xpath in the xml. From there, the JavaScript counts the number of characters from the beginning of the paragraph to where the user starts the selection. This creates the start point of the xpointer. A similar calculation is done for the end point of the region.

This creates a number of problems. First, not all portions of the article are easily annotatable because of the method by which the caluculations are done. If an area is created by pulling elements from different parts of the XML (such as the citation block area), it becomes difficult to map locations back to the XML. Additionally, if the elements are put in the HTML in a different order from the XML and user selects across regions, the meaning of the annotation changes if the HTML ordering is changed subsequently. Also, the client side calculations are inherently fragile due to browser differences in the treatment of XML. Greater support in JavaScript for native xpath is coming, but it didn't seem to be fully there when we implemented this originally.

Note that the current implementation of the xpointer is non-optimal. We provide start and end, but nothing else, causing the xpointer lib to match on every character in the range. An alternative approach is for the front end to continue passing that information, and have the back end construct the same xpointer, but then apply it to the xml doc and then construct an equivalent xpointer with syntax that will allow for a faster match.

Admin Console

• Lists article XML files from specified directory
• When “ingesting," it calls a Topaz web service that parses the article and inserts it into the store along with pulling search fields
• Upon completion, app queries for the image objects and begins to resize and store them
• Finally, it generates the XML file to send to CrossRef, which is stored on disk
• Copies the zip file into ingested directory upon completion (Note, on Windows, it sometimes has problems relocating document, but everything is OK)
• When publishing an article, the generated XML file is deposited at CrossRef's server

The figure and table images are processed into three different PNGs. One is the same size, but saved as a PNG. The second is scaled to 600px in the largest direction. The smallest is scaled to 70px wide. Inline-formula, disp-formula, and chem-struct-wrapper tagged images are saved as the same size, but as PNGs. Recommend to use ImageMagick and the JMagick bridge as tests I have run indicate much better performance and image quality (though the file size of the images are larger) and it is able to handle CMYK and RGB images correctly.

CrossRef XML file is produced on initial ingest. Currently, no references are sent. Authors list, dates, pub title, figs, etc are sent. See source:/head/plosone/webapp/src/main/resources/crossref.xsl for exactly what is pulled. There is a whole pdf book on the format of the file at CrossRef's site. You can also login and view the test and production queues if you want to track things. Rich has the necessary info to do so.

Recommend to move the admin app into it's own separate application

Development Testing

In order to develop thing in parallel, the Java application developers tested much of the functionality without using a rich JavaScript based front end. To test the actions we developed some basic FreeMarker templates. They can be accessed from /admin/oldMenu.action and can be used to directly submit values to the action and check the return data. The security enforcement is done at the XACML level, though the launch page (oldMenu) does require you to be an admin. If you list the articles, you can also perform operations like querying for the annotations, rating the article, deleting annotations, etc.

In general, one should be able to test WebWork actions outside of a container. However, during the latter phases, of development, a switch was made that required making the code managing the web services session-aware due to PLoS ONE having to send a session to the ProtectedServiceFactory. As a result, the actions became dependent on a container. With the introduction of the OTM layer and security moving up to the application level, as well as the introduction of the dummy authentication layer, it should be possible to have the unit tests work properly again.

Caching

For PLoS ONE, we use OSCache. It is used in two different ways. One is set up as a filter on the web application. This one caches the article images, pdfs, etc. A cache hit means that the request never even gets to webwork. The key is based off the request, and the actual response stream is cached and sent back. This is configured by the oscache.properties file. A separate cache is maintained for the articles. This ia configured in the applicationContext.xml file of PLoS ONE. OSCache makes use of JGroups for reliable multicast communication. If you have problems with cache communications, the JGroups web page is a good place to go. May be worth upgrading to OSCache 2.4 to get the JMX and Spring integration so that this info can be added to the admin console. Note that OSCache does not send any objects across the wire. The only communication is cache invalidation messaging (and setup/teardown, obviously). I also never was able to get the JSP taglibs for OSCache to work correctly with FreeMarker, though I did not spend a lot of time on it. It would be useful for some portions of the site like the home page or even browse.

Because of performance issues, there is heavy caching with viewing of articles. The html output is cached, as is the list of annotations. The XML of the article should be cached (especially since it doesn’t change), however it became difficult to store the DOM document cleanly, and I ran out of time to implement the caching of the document properly. The output of applying the annotations to the document and the resulting application of the XSL is instead cached. Whenever an annotation is added or deleted to an article, the cache is cleared. The grouping abilities of OSCache is used so that we can send a cache invalidate method to all object belonging to an article.

Registration

Registration is a separate webapp that talks directly to the Postgres database. It creates new login/password pairs along with the current status of the user (active/not active, address verified/not verified). You can disable a user from logging in by changing the active flag to false. CAS checks for that property. When creating a new user or resetting email address or password, it attempts to send an email out. Make sure you have and SMTP server running on your local machine or that the smtp server is configured properly in nonJmxApplicationContext.xml. Verifications are token based and the token is overwritten in the DB if a subsequent request is made. The global_config.ftl file that has to be configured can be factored out into /etc/topaz. You can just add a template search location to go there first.

Dojo Packaging

In order to reduce the size of the Dojo payload, production is configured to point to compresssed versions of the Dojo libraries. The plosOne.xml checked in to the pacakges directory and put into /etc/topaz points to these files <file>/javascript/dojo/dojo_home.js</file> and <file>/javascript/dojo/dojo_plosone.js</file>. The reason for the two different files is because the home page uses far less of the Dojo libraries, and the resulting file is about 100K smaller in size. As a result, to reduce load times on the main home page, I elected to construct a special file for it. The rest of the site uses dojo_plosone.js. The files are built using the standard Dojo build utilities. Specifics can be found on the Dojo website. You should build using intern-strings and can optionally use strip-resource-comments. Consult the dojoRequire_*.ftl files to determine which Dojo packages need to be bundled into the build. Note that in development, to ease debugging, we use the exploded dojo.js configuration. Thus, the plosOne.xml checked into /WEB-INF/classes should point to /javascript/dojo/dojo.js and you should edit config.xml to use the local plosOne.xml, not the one in /etc/topaz.

Templating

PLoS ONE uses FreeMarker as its templating engine. Template engine means various things. For FreeMarker, it means simply a way output text given some input. For our purposes, we wire in to WebWork to output the contents of Java Beans. However, I didn't see a good way to create easily maintainable site-wide templates (from a design perspective). Something along the lines of Tiles, which wasn't supported at the time the application was built, but now can be used with Struts 2. As a result, I created a custom WebWork result called PlosOneFreemarkerResult. This basically allows you to define a template layout file in xwork.xml and have the results of your action go to that FreeMarker file and have the parameter templateFile passed to the page and inserted. In conjuction with the startup configurations classes which read the file plosOne.xml, the page is assembled with the defined css and javascript inclusions as well. See /javascript/global_js.ftl and /css/global_css.ftl. Finally, the result class also allows you to specify a noCache paramter in xwork.xml to tell it to set some HTTP headers to attempt to prevent the browser from caching the page contents. When the rpms were initially created, we didn't have time to customize everything to tune it for production performance. One of the items left off was the caching of freemarker templates. We should set the value to a high number since the templates do not change on production. You can set the template_update_delay property in freemarker.properties to a high value (in seconds) as part of the rpm build process. It should be low in development for obvious reasons.

JavaScript

Topaz Javascript

For the current javascript implementation for Topaz, the Dojo Toolkit is currently being used for its utilities and dialog widget. In order to use Dojo, the javascript needs to be included in the following order.

1. Dojo configuration object, djConfig.
2. Dojo build file.
3. Dojo.require commands that will tell Dojo which modules that you want to use.
4. Any custom javascript objects.
5. Initialization files.

Creating a custom widget for Dojo 0.4 and below: (Dojo 0.9 dijits (formerly widgets) do things differently) As an example, I’m going to use /javascript/topaz/widget/RegionalDialog.js. To create your own custom widget, first take a look at the widget that you want to customize. The widgets usually have a “base” class and a class that you actually call. At the top, declare your widget name and the namespace using a dojo.provide

dojo.provide("topaz.widget.RegionalDialog");

Then include the components you’ll need to build your widget:

dojo.require("dojo.widget.*");
dojo.require("dojo.widget.ContentPane");
dojo.require("dojo.event.*");
dojo.require("dojo.gfx.color");
dojo.require("dojo.html.layout");
dojo.require("dojo.html.display");
dojo.require("dojo.widget.Dialog");                     // for RegionalDialog
dojo.require("dojo.html.iframe");

Then declare your base class by specifying your class and followed by an array of objects that you’re extending. Finally, you start your method definitions. Unless you’re modifying every method in the original widget, just define the methods that you want to override.

dojo.declare(
            "topaz.widget.RegionalDialogBase",
            [dojo.widget.Dialog],
            {
              …some code
            }

For the class that you’re calling, use dojo.widget.defineWidget and specify the name of your widget class, an array of objects that you’ll be referencing, and then your method calls.

dojo.widget.defineWidget(
            "topaz.widget.RegionalDialog",
            [topaz.widget.RegionalDialogBase],
            {
              …some code
            }

In this call, note all the places where it’s making specific references to the base class and include these also.

You will also need to include some supporting files.

1. In the directory in which your custom widget, you need to include a file called package.js with the following code:

   dojo.provide("topaz.widget");
   dojo.widget.manager.registerWidgetPackage("topaz.widget");
   

2. In the topmost folder of your custom widget, in this case it’s /topaz, you’ll need to include a manifest.js file where you’ll map your custom widget to a name that can be called in the html. For any further widgets, you’ll only have to add to topazMap.

   dojo.require("dojo.string.extras");
   dojo.provide("topaz.manifest");
   var topazMap = {"regionalDialog":"topaz.widget.RegionalDialog",
       "someOtherWidget":"topaz.widget.SomeOtherWidget"};
   dojo.registerNamespaceResolver( "topaz", function(name) {
   return topazMap[name];}
   );
   

3. Add these lines first in the dojo.require section of your html header so dojo can find your custom pages. The two key elements are dojo.registerModulePath for every file path of your custom widget and dojo.require for every custom widget.

   dojo.registerModulePath("topaz", "../topaz");
   dojo.registerModulePath("topaz.widget", "../topaz/widget");
   dojo.require("topaz.topaz");
   dojo.require("topaz.widget.RegionalDialog");
   

4. Then you include your widget by creating a <div> with a custom attribute called “dojoType”. This is where you’ll use the name that you’ve specified in the manifest, eg., regionalDialog.
5. There’s an additional template you may have to include if you intend to change the template somehow and that will be stored in /javascript/topaz/widget/template.

That should do it.