dtSearch Implementation in ILG

Hi everyone,

Here is the recording from today's meeting to discuss the changes in the UI design:

GMT20210520-151708_Recording_1920x1050.mp4 127 MB • Download

As discussed, because it will very tight for us to complete and finalize the design across the different views within search, we'll aim to perform the dtSearch implementation in Sprint 4 between June 21st and July 12th. Rob and Radomir let us know if you foresee a problem with this timing, because I know Radomir mentioned that he might be taking time off after July 4th.

We also discussed and agreed to proceed as follows:

1. Melissa and Naomi will complete the modified design (user stories and wireframes) for presentation and discussion during the meeting scheduled for Thursday, May 27 at 8:15am PST.
   • During the meeting we will determine whether modifications to the HTML templates will be required.
2. Assuming the design is finalized next week, we will integrate a proof of concepts for the updated design into the Sprint 3 (May 31 - Jun 21) for the Browse by Topic search (SQL search).
3. Assuming the proof of concept is a success in Sprint 3, we will proceed with integrating dtSearch into the Browse by Keyword search in Sprint 4.

TologixILGDBIndexer.exe indexer-config-ilg.json

1. Not all results have matching "paragraphs". I suspect it's because some HTML documents could not be parsed (using the logic from the ISLG project), as I saw several such errors in the indexing log. 
2. Fields with multiple values and all concatenated by dtSearch. We should probably use some separator character(s) as we did for ISLG. (I could either do this in the indexer, or you can do it in the view.)
3. Filtering on field values containing more than one word or special characters doesn't work well with dtSearch. I guess most of the meta field values are keyed so we should also index their IDs and filtering should be by IDs. (As we did  in ISLG.) 

For example, maybe you could change the vw_DTSearch_Project_Document_MetaFields to have both value IDs and labels in separate columns, where multiple values are separated by |||. One such row could look like this:

image.png 8.41 KB • Download

Obviously I don't have proper ValueIDs for this example but I hope you understand. (I excluded ContentTypeDataId column as it's not indexed.)
Using this data, I'd create two separate fields, pmf33 and pmf33id containing values and value IDs respectively.

I hope we're on a good track. Let me know if you have any questions.

Hi Hiren , I'm having a hard time trying to understand some of conditions. For example:

1.  ProjectMetaFieldId(pmf3ids) = 3(Prject Name) - looks like you're trying to limit search to pmf 3 with numerical values (178, 215, 252). However, indexed values for pmf3 don't have numerical values - IDs like "AlbertaSchoolsASAPI", "Sharmilatestproject", etc.
2. For pmf2, for date ranges, you need to format dates in the YYYYMMDD format, the same as for sorting. You need to fix this in the DB view.
3. For tmf50, you have condition (Value=true AND Value=False). For a single value field this doesn't make sense as it will always give FALSE as the result.

Hi Hiren , here's an example filter, close to what you asked for, except I used IDs that appear in the sample content in order to get results.

{
    "searchRequest": "canada",
    "SearchType": "Boolean",
    "Stemming": true,
    "WordNetSynonyms": false,
    "Fuzzy": false,
    "Fuzziness": "1",
    "FilterStatement": {
        "type": "boolean",
        "Operator": "and",
        "clauses": [
            {
                "type": "boolean",
                "Operator": "and",
                "clauses": [
                    {
                        "type": "boolean",
                        "Operator": "and",
                        "clauses": [
                            {
                                "type": "match",
                                "field": "pmf3ids",
                                "values": [
                                    "AlbertaSchoolsASAPI",
                                    "Sharmilatestproject"
                                ]
                            },
                            {
                                "type": "match",
                                "field": "pmf3ids",
                                "exclude": true,
                                "values": [
                                    "145",
                                    "289",
                                    "326"
                                ]
                            }
                        ]
                    },
                    {
                        "type": "range",
                        "field": "pmf2ids",
                        "from": "20050817",
                        "to": "20210819"
                    },
                    {
                        "type": "match",
                        "field": "tmf50_1",
                        "value": "false"
                    },
                    {
                        "type": "boolean",
                        "Operator": "and",
                        "clauses": [
                            {
                                "type": "match",
                                "field": "pmf34ids",
                                "values": [
                                    "608",
                                    "22",
                                    "47"
                                ]
                            },
                            {
                                "type": "match",
                                "field": "pmf34ids",
                                "exclude": true,
                                "values": [
                                    "11",
                                    "67",
                                    "88"
                                ]
                            }
                        ]
                    }
                ]
            }
        ]
    },
    "SortField": "modifiedon",
    "SortOrder": "desc",
    "PageNum": 0,
    "PageSize": 20
}