Overview of indexing, and alternatives consider (discuss)

Morgan and Anil,

Sorry for the title on this thread! Couldn't come up with a better one on a Friday afternoon. ;)

The following is based on our call yesterday. I've summarized the workflow as it was described in ISLG and (proposed) for P3 with dtSearch/PDF-to-HTML conversion and followed it with some suggested reading on alternatives to dtSearch for DevIT to evaluate.

Workflow
When I refer to "application", it is meant for P3 and the ISLG-rebuild respectively. Not the current running version of the latter.

Missing above is any additional database activity that may take place that links any of the meta data (tags etc.) to the uploaded PDF and converted HTML. We touched on it briefly, but didn't fully clarify this process. The assumption right now is that the external Windows service referenced in 1. is also handling that workload (Anil, please confirm this assumption).

As mentioned in the call, a background process for item 1 is the best way to proceed. Anything that takes longer then 5-10seconds (even that is a big stretch) in a web application is not ideal and should be handled via a background process or a separate thread.

I mentioned this as an option for background job handling in .Net (.Net Core supported too) as a potential option.

Hangfire
https://www.hangfire.io/

It integrates pretty seamlessly into an application to fire off a thread handled outside of the web request that the main thread can poll to check on the status. Useful functionality if UX for the admin requires the final loopback post document conversion to HTML.

Another option is for the application to poll for the existence of the converted HTML file and the final existence is confirmation of the successful conversion.

The latter might be the best approach given the time investment in the network service that is monitoring the PDF folder. Potential risks could be that the existence of a file (or not) may not provide enough information to the application (and extension, the admin) if something went awry during the conversion.

dtSearch alternatives

This isn't meant as a slight on dtSearch, it is just some options we might want to look into further as there are a number of really good full-text search engines/services available that may provide additional features or, at the very least, drop the requirement for that second external interface for their DLL.

Elasticsearch
https://www.elastic.co/

Starting with likely the biggest option in this space right now. It's fully open-source, and can be installed on Windows (primarily meant for Linux, but they have a Windows installer and service). Support can be purchased from Elastic if required though.

It is built on-top of the powerful Lucene full text search engine but provides an easy-to-consume API in front of it that supports standard REST-ful actions against it, and returns JSON.

They provide a plugin interface which supports ingesting documents that will parse a number of document types (HTML, PDF, Word etc.) and index them similarly to how the dtSearch process works. It is event driven, not a folder monitoring search, so the application would have to send the document contents to the Elasticsearch API for it to index.

They also provide .Net clients that fully support the .Net Core standard so will work without requiring an additional application.

It supports clustering, and name-spacing is easy if you want to use the same Elasticsearch environment for both the re-built ISLG application and P3.

Solr
http://lucene.apache.org/solr/

Like Elasticsearch, Solr is built on-top of Lucene for it's full text search capabilities. Solr provides similar interfaces dependent on REST actions and can output responses in XML, CSV and JSON.

There are no official .Net libraries for Solr, but there are a few community options available that both support .Net Core natively.

There are a lot of similarities to the two products, highlighted in detail here. However, the largest difference that impacts P3/ISLG is that Elastic is a full-fledged company behind Elasticsearch while Solr is an Apache open-source product. There are third parties that will provide Solr support though.



For both of the options above, the Facet capabilities built into Lucene might be worth exploring as it provides some additional search options that a more traditional taxonomy (hierarchical, text-based) may not support, such as numeric/date-based data and can be derived automatically through indexing rather than assigned manually.

Another feature that in Lucene is Boosting. This allows the index definition to determine if certain data should be weighted differently in queries. For example, text provided in heading would bubble up higher in search results than similar text in document bodies (a very basic example).

An interesting project came up while looking into this:

Ambar
https://ambar.cloud/

Specifically, as it was developed as an alternative to dtSearch.

It's relatively new, largely an amalgamation of various open source tools (redis, mongo) and we have never used it for any project, and likely won't work on Windows. It's not a recommendation to look into them as an alternative, but they chose Elasticsearch as the primary text-search engine that their tools rely on.

There are other options in this space but these Lucene-based products are really the big open-source players in this space, and the ones we have the most experience with. Good support for .Net, with support options available, and both have very good APIs.

As discussed though, the difficulty will be translating the search parameters that the application currently relies on to the APIs that these tools provide (or the abstractions provided by the various Nuget packages) and assessing what the effort will be to port existing logic over for both the request and response handling in the application.

In any event, there's a lot here to look through and I hope that this helps in some way. If there are any questions, or a follow-up call would be beneficial, let us know.

Have a good weekend!