Ingestion should be standalone application.

This will require a redesign and rework of ingestion. Are you sure you want this at critical level as it will take away resources from other 0.9 features?

since 0.7, we have down time every week on ingest and publication of articles and susanne has to spend hours and hours dealing with timeouts, errors, etc. and i have to spend hours handling publication.

is this going to get better or worse when we go to 0.9 and are publishing for 2 journals instead of just one?

we can make some improvements to the process by having a separate admin vhost if you all can help with that (see #521)

IMO, anything that causes consistent outages on a production site is critical and needs to be fixed now.

also, this is a piece of the topaz infrastructure that has been bad from the start - nothing good can come of waiting to fix this as we add more journals, more articles, more features.

i'll ask rich to comment - i really can't tell you whether critical bug A comes before critical feature B.

0.9 features and NTD launch are the priorities....

Please take a look at my #521 comment. The essenece there is to ensure admin pages get redirected to a single Publishing App server. You guys probably know how to do that much better than I do. If that works, then I would prefer finishing up NTD and one good new feature for 0.9.

#521 makes it possible for me to push all the production work back on susanne - but it doesn't fix any of the underlying problems.

ingest will still cause long load times. we'll still have to take the site down to publish. it's an improvement to process in many ways but no fix for this issue.

Sorry Russ. I confused the thread. You are right. #521 will not improve the load time. The reason ingestion is taking a long time is the image transformation being done as part of the ingestion process. If that is becoming painful, we have to move transformation as part of standalone tools. Let me know if you want this as 0.9 as it requires a bit of work.

this issue is not about ingest taking a long time.

it's about the fact that, during ingest, the site is so slow that we get nagios alerts saying the site is down.

the slowness is intermittent - it doesn't happen on every article. i don't know what causes it. the behavior is similar to the other issues we've had involving mulgara locking.

i don't know if that's related to image transform or not.

i thought we were going to full command line ingest for 0.9 - i certainly would suggest that we do - but it's unclear to me whether that has any bearing on this problem since i imagine mulgara does the same thing for command line or browser based ingest.

Replying to russ:

this issue is not about ingest taking a long time. it's about the fact that, during ingest, the site is so slow that we get nagios alerts saying the site is down.

Please keep in mind that the image transformation is taking place on the server. Our logs seem to indicate that this is taking the most time and I am *assuming* image transformation is also overloading the CPUs.

the slowness is intermittent - it doesn't happen on every article. i don't know what causes it. the behavior is similar to the other issues we've had involving mulgara locking.

That might be based on number of images within the article.

i don't know if that's related to image transform or not. i thought we were going to full command line ingest for 0.9 - i certainly would suggest that we do - but it's unclear to me whether that has any bearing on this problem since i imagine mulgara does the same thing for command line or browser based ingest.

The command line ingest has been on the wish list and not assigned a priority. Please confirm with Rich and let us know.

Milestone Bugs deleted

i'll check cpu usage on next ingest - but i think it's unlikely that this is the cause - it really feels like mulgara is blocking and imageMagick isn't that big of a hog in my experience.

i'll update after next week's ingest.

One more thing...Mulgara allows only one write transaction at a time and others block waiting for the write to finish. If the ingestion takes a long time, it will keep a write transaction open blocking others making it appear to be slow.

hmm. so is it the case that mulgara opens a write before imageMagick starts, and keeps it open until imageMagick finishes?

i'll check and see how long imageMagick is taking as well...

That is a possibility. I don't know the precise sequence of operations in ingest.

Amit, the problem is not the image-transformations. Yes, they take time, and yes, the use cpu, but they don't cause the site to block. What causes the site to block is the article cache being cleared and both plosone's having to load all articles and associated objects, which takes several minutes, and which blocks all other activity that needs to talk to mulgara. And no, moving the image-resizing and other stuff out into a standalone app will not help at all here.

The only things that will help are A) to optimize the home-page's and browse-page's queries (instead of loading all articles as now), and/or B) for mulgara to support multiple simultaneous read transactions. Another option would be to avoid blowing away the caches and instead support incremental updates of the internal data structures.

hmm...as i understand it the article cache is only cleared on PUBLISH, this problem is happening on INGEST, i don't think it's the same thing...

rich says nothing takes priority above the enhancements necessary for 0.9 - so as much as i want these things fixed soon i understand that they may or may not happen in 0.9.

Replying to amit:

One more thing...Mulgara allows only one write transaction at a time and others block waiting for the write to finish. If the ingestion takes a long time, it will keep a write transaction open blocking others making it appear to be slow.

Actually, Mulgara allows only a single transaction at a time. Period. Independent of whether it's read-only or read-write.

Also, there are many transactions involved during ingest: one for the basic ingest (processing the zip, loading the mulgara, fedora, and search), and one for each resized image. The basic ingest takes about 50 seconds; the other ones are much shorter (10 seconds).

Replying to russ:

hmm...as i understand it the article cache is only cleared on PUBLISH, this problem is happening on INGEST, i don't think it's the same thing...

Ooops, my bad, you're right. Sorry, I see now that you're talking about a different issue with ingest. In that case it must be those 50 seconds for the initial ingest. Note that most of that time (40+ seconds) is spent loading mulgara, fedora, and search, so moving the zip processing out into a standalone app still won't help. It looks like the Fedora loading is taking the most time, so moving that outside the mulgara transaction would help.

i'm going to assign this back to amit.

it sounds to me that this might be an issue even after we move to command line ingest if the main issue is that mulgara is waiting for fedora to finish.

post 0.8, perhaps someone can invesigate further?

Btw., didn't you have the same issue with 0.6? This part of the code hasn't really changed - it just got moved from topaz to plosone.

i don't *think* we had this problem with 0.6, but if you remember we did then you're probably right.

i guess i agree that this is not critical, but it's still a big problem.

let's look into moving long operations (loading fedora, imageMagick) outside of mulgara transactions.

I just noticed why this problem wasn't there in 0.6, but started with 0.7: we used to have a transaction covering just the mulgara inserts, but now the transaction scope covers the whole ingest, including fedora and lucene (but not the image scaling).

Replying to ronald:

I just noticed why this problem wasn't there in 0.6, but started with 0.7: we used to have a transaction covering just the mulgara inserts, but now the transaction scope covers the whole ingest, including fedora and lucene (but not the image scaling).

What was the reason for the shift?

Updated because of r4040.

Btw, not creating all those unnecessary fedora objects during ingest should help a bit here. Over a quarter of all objects in fedora are the category objects, and the RELS-EXT streams are not needed either. Because this stuff is completely unused, we can just stop creating it on ingest, i.e. no data migration or any other changes are needed.

Ingest will remain part of the Admin console and will be significantly reworked:

#722, #550, #713 #675

closing this ticket.

I don't get this. What is the reason for closing this? Is there another ticket covering this issue? I am worried that issues will get dropped as it exists in our minds and not on Trac.

Duplicate of #848.