Search Implementations

Hello all,

Further to my message above, Rob and this team are going to get involved in building our front-end searches for ISLG and ILG. They will begin development after the UI features and the SQL databases are complete in October. At that time, Contegra will be able to thoroughly review the new database structures and understand the scope of work needed to complete the necessary tasks.

At this point, we anticipate Contegra performing the following:

Custom Indexer Service: Contegra will create dtSearch indexes containing both the full text and the relevant SQL fielded information, which will include:

1. a command line interface to create a new index from scratch; and
2. the ability to check the file system directories for new or changed documents and process updates.

1. creating a web service that will handle all search requests from the client applications (ISLG and ILG);
2. allowing the client to communicate with the web service using REST-like interface, returning data in JSON format; and 
3. creating a web-service based on .Net that uses IIS. 

1. setting up requirements for periodic merging of updates and optimizing indexes;
2. setting up index backup requirements; and
3. creating protocols for error handling.

Please let me know if you have any questions about the above.

In the interim, Jitesh please review and let us know if you foresee any problems with integrating these plans into your existing development timelines.

Thanks,

Morgan 

Hi Rob ,

In preparation for the call on Thursday, Melissa has put together a document that outlines the requirements all of the searches that will be available on the subscriber side of the new ISLG application: https://docs.google.com/document/d/10TP4xS4YUgmnznIUI2pzzMA2HOZu1FPm8dE7zmgudtA/edit#heading=h.tuh7ytex0a7y.

The document is broken down into four categories of searches:

1. Search
2. Research Tools
3. Document Library
4. Other

For each search, Melissa has provided screenshots and links to the relevant wireframes. She has also pulled all the acceptance criteria from the relevant user stories. This should give you all the detail you'll need to understand the requirements of the various searches.

For the purposes dtSearch, the only search that performs searches of the document texts is the Full Text Search (and perhaps the Global Search via the Full Text Search), and thus I believe is the only search that will require dtSearch. dtSearch is also used for the Subject Navigator in the old application, because we integrated the Boolean and linguistic options available through dtSearch. However, it appears we discarded these requirements in the new application ( Melissa do you know if we did that purposefully or is this an oversight in the requirements?)

Let me know if you have any questions or concerns. I would be happy to hop on a call in advance of the call on Thursday to explain anything in the document, so that we can focus on how to optimize all the relevant searches.

Thanks,

Morgan

Morgan  

We went back and forth on this a couple times. The intention was to simplify/standardize the search across research tools. That being said, we did end up modifying the behaviour for the Subject Navigator and integrating boolean and linguistic tools is certainly possible.

We could incorporate the following criteria:

I can enter a keyword search that is powered by the dtSearch search engine

• From the SN search input, I will be able to type in a keyword, and run a search
  • Keyword search is performed by entering a term or terms and hitting [Search] to submit
  • I will be able to search using three different methods
  • I can select my keyword search method using a radio button:
    • I can select 'Any words':
      • Searches 'any words' typed into my search field 
        • An "any words" search request consists of an unstructured natural language or "plain English" query. In a natural language search request, words such as AND and OR are disregarded. Use quotation marks to indicate a phrase, + (plus) to indicate a word that must be present, and - (minus) to indicate a word that must not be present.
    • I can select 'All words': 
      • Searches 'all the words' typed into my search field
        • An "all words" search is like an "any words" search except that all of the words in the search request must be present for a document to be retrieved.
    • I can select 'Boolean' (default selection):
      • Searches keyword(s) entered into search field using 'boolean' logic
        • A boolean search request consists of a group of words, phrases or macros linked by search connectors such as AND and OR to precisely indicate the relationship between them.
  • I will be able to enhance my search by selecting various linguistic aids to use
    • I will be able to select 'Stemming'
      • This will search word and its modifications that have the same stem e.g. like, likely, likelihood
    • I will be able to select 'Synonyms' 
      • This will also search words that are synonyms of the search term eg. discrimination, bias, favouritism. 
    • I will be able to select 'Fuzzy typo' (default selection - 1 character)
      • This will include words that differ in their spelling from the search term by the number of letters selected, e.g favor/favour
      • I will be able to the number of letters to include in the fuzzy type by clicking from 1-10. This option will be disabled if the above check box has not been selected. 

Hi everyone,

Following up on the call this morning with Ketan , Jitesh , Harsh , Rob and Radomir , we've decided to address the search implementation issues as follows:

1. Rob and Radomir will build the customized indexes necessary for all searches that utilize dtSearch, which will include the following
   1. Full Text Search
   2. Subject Navigator Search
   3. Document Library Searches
      1. Treaties & Rules
      2. Dispute Documents
2. All remaining searches will be performed using SQL database search, which will include the following:
   1. Research Tools:
      1. Jurisprudence Citator search
      2. Article Citator search
      3. Publication Citator search
      4. Terms & Phrases search
   2. Research Notepad
   3. Document Comparison
3. Ketan , Jitesh and Harsh will assess the requirements  and projected performance of the SQL searches above, will report back if any of these searches will take more than 2 seconds to produce results, and then we will assess whether further customized indexes are required.
4. Global Search will be performed by sequentially running searches across all the applicable searches, which will include SQL and dtSearch. However, if the performance of the Global Search is insufficient, we will explore the option of building a customized dtSearch index for the Global Search

As a first step, I need to confirm some paperwork with Rob . When I confirm that is complete, Ketan , could you please provide Radomir and Rob with the database views they requested over the call. As well, could you please provide them with remote access to the ISLG application server.

Lastly, for ILG, we are likely to adopt of a similar approach of creating a customized index for the dtSearch keyword search, but I would like us to finalize things in ISLG before we start that work. Therefore, Ketan , Jitesh and Harsh , please defer all work concerning search features in ILG until we have developed things further in ISLG. However, if you find that deferring this work is disrupting progress in ILG, please let me know, and we will assess and adjust as necessary.

Please let me know if anyone has any questions or concerns.

Thanks,

Morgan 

Hi Harsh ,

Questions like this are answered in the document produced by Melissa here: https://docs.google.com/document/d/10TP4xS4YUgmnznIUI2pzzMA2HOZu1FPm8dE7zmgudtA/edit.

In the acceptance criteria for Subject Navigator, the keyword search will be performed for the following fields:

• Branch Text fields for all Subject Navigator branches
• For any Dispute Document associated with a Subject Navigator branch, the applicable:
  • Dispute fields: Respondent State, Case Name, Case Number and Special Search Terms
  • Dispute Document fields: Short Title and Full Citation

Therefore, it would include the Dispute Document Full Citator (and other fields related to the document), but not the cited paragraph number or text. Going forward, it's very important that you consult this document and the applicable user stories to understand the requirements of each search.

Thanks,

Morgan

cc: Radomir Rob  

ExampleCustomModel.zip 1.27 KB • Download

Harsh , as discussed, I'm sending you an example attached how you can deserialize search engine response to a custom model. After deserialization, you can simply access fields as in: 

response.Results[0].Data.Filename

Hi Rob and Radomir ,

As discussed on Friday (14/08)'s call, We have added HierarchicalParentIds column in Subject Navigator SQL view.

• vw_SubjectNavigatorSearch

We have updated the SQL view on ISLGRebuild databse on  10.68.138.11 server.

We have remian ParentId column as it is and added new column HierarchicalParentIds to get multiple parentId in comma separator.

The HierarchicalParentIds column contains multiple ParentIDs with comma separator as per following screenshot.

SN_SQLView.PNG 37.7 KB • Download

Please let us know if you have any concern.

Harsh I have one more indexer update for you:
https://www.dropbox.com/s/riuyasnjn9rv7ll/DBIndexer3.zip?dl=0
It includes 3 dtsearch data files that define some indexing behavior (e.g. noise words). The code was updated to load them from the application folder. (This change was needed to fix behavior where single letter terms were not indexed so, for example, document ID with value "1" could not be found.)

Next, I implemented support for finding and returning parent nodes listed in the HierarchicalParentIds column. The search response payload now includes parents field witch is the same list of documents. Sample response looks like this:

{
    "totalResults": 7,
    "results": [
        {
            "fields": {
                "id": "35",
                "branchid": "35",
                "branchtypeid": "3",
                "branchname": "Test new subject",
                "Filename": "db://vw_SubjectNavigatorSearch#Id=35",
                "parentid": "20",
                "documentid": "15",
                "contenttypedatamasterid": "29",
                "disputedocumentshorttitle": "OT/0001/03 - NJ Test Case AF-0005-01 Destination Code - 31/08/2020 - English",
                "shorttitle": "NJ Test Case AF-0005-01 Destination Code",
                "fullcitation": "NJ Test Case, AF-0005-01 Destination Code, 31 August 2020",
                "casename": "NJ Test Case"
            }
        },
...
    ],
    "parents": [
        {
            "fields": {
                "id": "27",
                "branchid": "27",
                "hierarchicalparentids": "1,27",
                "branchtypeid": "3",
                "branchname": "Abuse of Process",
                "Filename": "db://vw_SubjectNavigatorSearch#Id=27",
                "parentid": "1"
            }
        },
...
        {
            "fields": {
                "id": "1",
                "branchid": "1",
                "hierarchicalparentids": "1",
                "branchtypeid": "3",
                "branchname": "A",
                "Filename": "db://vw_SubjectNavigatorSearch#Id=1"
            }
        }
    ]
}

Hi Harsh ,

I'm still working on the search service changes - you should have it tomorrow. In the meantime, here's the JSON structure that you can use to send your query:

1) Boolean queries nesting:

{
	"type": "boolean",
	"operator": "and"
	"clauses": [
		{
			"type": "boolean",
			"operator": "or"
			"clauses": [
				...
				another boolean or match type
			]
		}

	]
}

{
	"type": "match",
	"exclude": false,
	"field": "Field_100",
	"value": "3"
}

{
	"type": "match",
	"exclude": false,
	"field": "Field_100",
	"values": ["3", "7", "14"],
        "operator": "and"
}

Hi Radomir ,

We have following operators which we need to include service while we pass filter data in json file.

• is set
• is not set
• is equal to
• is not equal to
• is after
• is before
• is between
• is not between
• starts before
• starts after
• ends before
• ends after
• lasts more than
• lasts less than
• is greater than
• is less than

Hi Radomir ,

Following, I have added discussed point as per todays call.

1) Re-indexing issue is resolved
2) As per your recommended, We have merged query and view in one Single Query for all data. so now there is no need of second view. Following, I have mentioned query name.
3) We have added Row number in query as unique identifier. (Column name RowId)

The Stored Procedure Name : FE_MetafieldwithValueDynamicFTS. You can find on ISLGRebuild databse on our server.

Now, There is no need of second view. We get all data from above mentioned query.

For Sorting, There are  following 4 field you can use from query.

• Relevance (default) - relevancy criteria based on current functionality of "hit count"  - Depended on hit count from your side
• Document Name (A-Z) - Column name (FullCitationText)
• Newest First - Column name (SortingDate)(We have provided this column in  yyyymmdd format)
• Oldest First - Column name (SortingDate)(We have provided this column in  yyyymmdd format)

Hi Harsh ,

In the https://1drv.ms/u/s!AugzRBG6eTFwg4sJwygyvkWxOQnMWg?e=gfh2JQ
you can find a new folder with FTS indexer and the web service update.

Indexer config file indexer-config-fts.json specifies paths to two different indexes: 
- IndexDir (as before) is the main documents index
- ParagraphIndexDir is for the index containing paragraph-level data.
Both indexes are created in one run.

Both indexes should be added to the web service appsettings.json as in:

    "Tologix": {
      "FullTextDocsIndex": "y:\\contegra\\tologix\\test-index\\tologix-index-fts\\",
      "FullTextParasIndex": "y:\\contegra\\tologix\\test-index\\tologix-index-fts-paras\\",
      "SubjectNavigatorIndex": "y:\\contegra\\tologix\\test-index\\subject-nav\\",
      "DisputesIndex": "y:\\contegra\\tologix\\test-index\\tologix-index-disputes\\",
      "DisputeDocsIndex": "y:\\contegra\\tologix\\test-index\\tologix-index-dispute-docs\\"
    },

Hi Harsh ,

Paragraph extraction was implemented for the HTML file format you send me and highlighting for that worked fine on my end. This HTML had:

• <div> elements with class "paraBullet" containing paragraph number, and
• elements with class "ParaText" containing paragraph text.

Hi Radomir ,

Hope you are doing well.

We are start looking into the FTS module bug and for 2 bug we performed following scenario.

Scenario 1 :

• Enter only  keyword test 
• Click on Search button 
• The API provides the result with data where language display English

• Enter only  keyword test 
• Apply Filter language English
• Click on Search button 
• The API doesn't provide any data 

Hi Radomir ,

Following, I have mentioned the bugs description and issue which Industrail team raised.

1 ) issue no : 20579 - Search results with no pinpoint references

Steps to reproduce:

• Go to full text search
• type 'interpretation' in the search input field
• submit your search

• Go to Full Text Search
• Make a selection in the 'Language' field (ie: French)
• Scroll down and select [Submit search]

• go to full text search
• Type the term 'test' in the search input
• Select 'English' from the language field
• submit the search

• Go to FTS
• Add any filtering parameters but not a keyword and press "search"

• Go to FTS
• Set search type to "boolean"
• Enter yellow OR blue in the keyword search and press search

• Go to FTS
• Enter term "blun" into keyword search
• Set fuzzy typo to include 1 letter
• enter search

• <div> element with the paragraph number had "paraBullet" class, and
• the following <div> element with the paragraph test had "ParaText" class

Hi Radomir ,

There are 2 bugs produced by Industrial Team. Please look into this.  

To check both issues you can use ISLGRebuild database on server. and all PDF & Html documents you can find from following path : E:\ISLGRebuildDemo\wwwroot\Documents

21136 : Search > All words > no pinpoint references

Steps to reproduce:

• go to FTS
• enter the terms 'greek and september or tribunal' into the search field
• select 'All words' as the search type
• submit the search

Result: Search results cards  do not include pinpoint references

Expected: A paragraph link must exist for each paragraph in the document where at least one of the  keywords from the search was found

image.png 323 KB • Download

21138 : Search > Keyword not highlighted


Steps to reproduce:

• Go to FTS
• eneter the term 'like' into keyword input
• submit search
• from search results, find a result with pinpoint references
• Select a pinpoint reference to preview excerpt

Result: No keyword is highlighted in the result. The keyword does not seem to appear in the paragraph.

Expected: Only paragraphs with keyword appearances will be available as pinpoint references to preview excerpts. The keyword must be highlighted in the excerpt preview.

image.png 317 KB • Download

Hi Radomir ,

We need to discuss following bugs with you on skype call. Please provide your confirmation. We are available between 11:00 AM to 6:00 PM IST.

21138 Search > Keyword not highlighted

Steps to reproduce:

• Go to FTS
• eneter the term 'like' into keyword input
• submit search
• from search results, find a result with pinpoint references
• Select a pinpoint reference to preview excerpt

Result: No keyword is highlighted in the result. The keyword does not seem to appear in the paragraph.

Expected: Only paragraphs with keyword appearances will be available as pinpoint references to preview excerpts. The keyword must be highlighted in the excerpt preview.

image.png 274 KB • Download

21217 : Preview excerpt > must not be displayed if not keyword was included in query

Steps to reproduce:

• Go to FTS
• Submit a search with no keyword 

image.png 342 KB • Download

21218 : empty results appearing in search

• Go to the subject navigator
• expand the search options accordion below the search bar
• select 'All words' option
• in the search bar enter the terms 05 bridgestone
• hit enter

• terms that match the search will be highlighted within the results

image.png 353 KB • Download

20613 : Search > Fuzzy Typo not working (In Subject Navigator Index)

• Go to FTS
• Enter term "blun" into keyword search
• Set fuzzy typo to include 1 letter
• enter search

• Go to SN
• Set search setting for fuzzy typo to 2
• type in "awurd"

Morgan dtsearch has confirmed that the 1-10 fuzzy designation is not based on a letter count. 

The correspondence between search fuzziness with the default 1 to 10 setting and character discrepancies is not "1 to 1"

However, you can fine-tune fuzziness "manually" with the % character. Please see pages 42-43 of https://support.dtsearch.com/faq/dtSearch_Desktop.pdf for more on this.

Hi Radomir ,

The Subject Navigator Indexing are generated now. But, the Dispute indexing not generated.

• indexer-config-disputes