ILG Document Collection Upload and HTML Coding

Hi Morgan ,

I have discussed with Ketan and we are doing planning to start ILG document collection upload and HTML coding. can you please update as below.

1,  Before we starting development of Batch process, we need to discuss so please provide your time to discuss for same.

2, We can start PDF quality report as per we have done in ISLG, so can you please provide list of documents ( Source files ) in order to generate PDF quality report.

Hoping to get source of documents today so we will plan to start as soon as possible. 

Thanks,
Jitesh

Hi Jitesh ,

How about we do a call on Friday at 8:15am Vancouver time? I'd Paul and Katrina to join the call as well.

Here is a link to the source documents: https://islg.egnyte.com/fl/TmxP3c15v8. Katrina , can provide you with more direction as required.

Thanks,

Morgan

Hi Morgan ,

Ok, Friday at 8:15 am Vancouver time is fine for us. We will download documents from above provided url and will start to generate PDF quality report.

Thanks,
Jitesh

Ok. Sounds good, Jitesh . I'll send out a calendar invite shortly.

Morgan

Hi Jitesh , Ketan and Hiren ,

Here is the recording of the meeting earlier today:

2020-07-03 08.23.06 ILG Data and Document Upload 84429801576.mp4 123 MB • Download

Thanks,

Morgan 

Hi Jitesh ,

An issue we didn't discuss during the meeting was where we are uploading all the data and documents. Because we still are working on the SME trial. My suggestion is that we upload all http://dev.infrastructurelawguide.com/, and then sync the data with http://www.infrastructurelawguide.com/ after the SME trial is complete.

What are your thoughts?

Thanks,

Morgan

Hi Morgan ,

Yes, we will first upload data and document on Dev than we will sync into production.

Thanks,
Jitesh

Ok. Great. Thanks Jitesh .

Morgan 

Hi Morgan ,

Please find attachment of ILG PDF Quality Report. We have added one column for UIN number as per your need. We have generated first 1444 PDF files quality report  and remaining files quality report is in process.

ILG PDF Quality Report.xlsx 137 KB • Download

Thanks,
Jitesh

Great. Thanks Jitesh ​.

I'll review this with Katrina ​ tomorrow and we'll put together the first batch of documents that will require manual PDF to HTML conversion.

Thanks,

Morgan

Hi Morgan ,

I have proceed all ILG documents to generate PDF quality report. We are getting following types of files after proceed all files.

image.png 8.83 KB • Download

Please find attachment of PDF quality report, you can find all types of files in this report.

ILG PDF Quality Report-09-July-2020.xlsx 274 KB • Download

Ok. Thanks Jitesh . I'll take a review, consult with Katrina and we'll sort out the issues you've identified above.

Morgan

Hi Morgan ,
Hope this is okay but I took a look at the issues and resolved most of the duplicate UIN number issues. The only project that I didn't fix was the Presido Parkway as that was one of the initial prototype projects.

Jitesh  
As a heads up for the Royal Inland Hospital (Sr. No. 1839), I had to update the UIN numbers for documents BC/0018/0043 - BC/0018/0047.

Hi Jitesh and Ketan

Katrina is going to review and resolve all the issues you've identified above, and when it's complete (at the end of next week), we'll provide you with an updated list of documents for manual coding in the ILG Manual Coding Report: https://docs.google.com/spreadsheets/d/1aS8_8W2aYmwRtveoVkNNRDu30ATxBkWlpNdEk9-ftBA/edit#gid=597349052. 

In the meantime, let us know if you have a questions or issues concerning the uploading of the data from the ILG Project and Document Master List.

Thanks,

Morgan

Hi Morgan ,

In ILG documents, we have need to confirm following cases from you whether it will need to convert html when converting manual conversion.

1,  Form   : We are getting form in PDF UIN number AB-0013-0005, in that PDF contains form so we need to generate or display [REDACTED], I am asking because form content checkbox, field and underline for input so please confirm.

image.png 47.6 KB • Download

2, Paragraph bullet not exist content: Ex . UIN : AB-0013-0022

In this case, Many PDF contains bullet without content so what we should display there for content. It should be empty or some REDACTED need to display in this cases. Please confirm.

image.png 24.5 KB • Download

3, Some Text found written manually.  UIN : BC-0001-0001

We have found extra text written by pen, so we need to consider those text or not. Please confirm.

image.png 61.6 KB • Download

Thanks,
Jitesh

Hi Jitesh ,

Could you please re-post the screenshots above, the images are extremely small and it's difficult to see.

Thanks,

Morgan

Hi Morgan ,

I have re-attached screenshots, Please provide your feedback.

1, Form

image.png 122 KB • Download

2, Paragraph Bullet without text

image.png 31.8 KB • Download

3,  Some Text found written manually.

image.png 183 KB • Download

Thanks,
Jitesh

Hi Jitesh, 

Please deal with the issues above as follows:

1. Forms: Please represent the forms as best as possible, but check-boxes can be represented as bullets.

2. Paragraph Bullet without text: It's difficult to understand exactly what's going on here, but please include the bullet number once with [REDACTED] afterwards (s.25  [REDACTED]).

3. Some Text found written manually: disregard all written text.

Thanks,

Morgan 

Hi Morgan ,

Before We start data upload activity, Can you please tell me all existing data related to Project and document should be remain as it is or removed and insert fresh data from batch process as per spreadsheet provided? like Meta fields values, List etc.

I have found dropdown list prepared by Katrina for Industry sector, Industry subsector etc.  Also I have found state/Province/Territory Meta Field's list are not proper maintained all values contains multiple comma ( , ) so please tell me we need to removed all list and do by batch process or need to insert only list value those are not present in the list.  Please provide your feedback.


Thanks,
Jitesh

Hi Jitesh ,

Katrina is going to do a final review of the master list, and post an updates version later today. However, to answer your questions above:

Thanks,

Morgan

Hi Morgan ,

Thanks, We will wipe all data from Dev and insert fresh data as per spreadsheet.

I know multiple value indicating with semi colon in spreadsheet but i am asking in production site "state/Province/Territory" Meta Field's list are not proper inserted, when you select item than it will contains multiple comma ( , ) so we need to correct those items in a miscellaneous list.

Thanks,
Jitesh

Hi Jitesh ,

Ok. I see what you mean. There appears to be a problem with the  US States and Territories list that is causing the problem. To resolved it, I suggest wiping these list, and uploading fresh versions. Here is the data you can use for the lists:

Canada Provinces and Territories.csv 252 Bytes • Download

Countries.csv 2.2 KB • Download

Industry Sectors.csv 87 Bytes • Download

Industry Subsectors.csv 1.4 KB • Download

US States and Territories.csv 1.33 KB • Download

The rest of the lists will be populated by the applicable entries within the master list that Katrina will be providing later today.

Thanks,

Morgan

Hi Jitesh ,

My apologies for the delay but I have updated and finalized the master list.

ILG Project and Document Master List - UPLOAD.xlsx 7.4 MB • Download

Thanks, 
Katrina

Hi Morgan and Katrina ,

Thanks for updating finalized the master list and we will consider attached spreadsheet by Katrina . But still we have some questions and suggestion in spreadsheet.

Project Document:

1, For Document, We have not found any Meta fields added in application based on spreadsheet provided in order to identify field type ( dropdown, textbox, etc )

2, I think, following fields not need to consider when uploading data into database for project document. Please confirm.

I am request you to please mention only those column which are need to consider for data upload into database. Also please create Meta fields for project document in the application.


Project :

In application, there is field set "Contractor(s)/Subcontractor(s) (Role)" contains three Meta fields like D&B contractor, D&B Design and D&B construction but in spreadsheet contain column name "Contractor(s)/Subcontractor(s) (Role)" and value contains with Semicolon ( ; ) separator and it found four (4) types of values there but you have only created three (3) meta fields so where to do add value for "O/M contractor".

Second, I am suggesting please create separate column for "Contractor(s)/Subcontractor(s) (Role)" values as per you have did for industry, like Industry sector , Industry sub sector you have created separate column in spreadsheet and in application all exists in "Industry" field set. Please follow same "Contractor(s)/Subcontractor(s) (Role)" so we can do proper batch processing.

For example

In application :

image.png 23.5 KB • Download

In Spreadsheet :

image.png 18.9 KB • Download

Please provide your feedback.

Thanks,
Jitesh

Hi Jitesh ,

Here is updated version of the master list:

ILG Project and Document Master List - UPLOAD V2.xlsx 7.19 MB • Download

We have removed all columns that are not relevant to meta fields within the application (except column F (Link to Document) in Project Documents in case this information is needed).

For the issue concerning the Contractor(s)/Subcontractor(s) (Role) fields, we have reorganized the master list separating the entries into the respective fields:

image.png 120 KB • Download

Also, I adjusted the fields in dev.infrastructurelawguide.com so that all the Contractor(s)/Subcontractor(s) (Role) fields pull from the same Contractor(s)/Subcontractor(s) List. This means that that same values should be used to populate all four fields, and that duplicate values between fields should be consolidated. For example, you'll see the value "EllisDon" appear in the "D&B Construction", "D&B Contractor(s) and "O&M Contractor". These are all referring to the same "EllisDon" value.

I also added the "External Sources" field to the application, which will be populated by column AL (External Sources) in the Project tab. Note that multiple URLs are separated by ";" and should be treated as separate values. Also, we did not include data for the "Display Text" field. I assume if this isn't included in the URL field, it will just display the URL.

Thanks,

Morgan

Hi Jitesh ,

I writing to let you know that we discovered an issue with a number of documents in the master list above that are currently being reviewed by Katrina , which will result in a further modification of the master list and the documents currently undergoing manual PDF to HTML conversion.

We have discovered that some documents are very large (500+ pages and more than 100MB). The large size is caused by the consolidation of project agreements and schedules into one document. We have decided to subdivide these documents into separate documents, and integrate them as separate entries in the master list. Note that this may affect some of the documents that are already undergoing manual PDF to HTML coding.

Katrina should have the updated master list complete by tomorrow, but I wanted to let you know if advance in case you and Hiren want to pause further work on the migration until you have the updated master list.

Thanks,

Morgan

Hi Morgan ,

We are doing final inserting data upload for Project and project document and verifying in local system and we have following question so please provide your feedback.
 
1, Can you please tell me what should be PDF name in project document? you have not provided PDFName column.

2, When inserting project document data which status should be for document. I think, it should be "Edit in Progress" and user must be upload PDF and HTML in future in order to see/update document data in application. 


Thanks,
Jitesh

Hi Jitesh , 

Just to clarify, when you say that the PDFName column is missing are you referring to the Master List or the ILG HTML Manual Coding spreadsheet? 

Thanks, 
Katrina

Hi Katrina ,

No, i am taking about data upload of project and project documents.

Thanks,
Jitesh

Hi Jitesh ​,

You will have to retrieve the PDF filenames based on the provided Egnyte URL links. At the same time, the UINs are contained in the filenames, so you could use that to identify the filenames as well.

I'd like all the data and documents published and visible of the subscriber side of the application. Please have them uploaded in the Published state and then we will selectively edit documents that require tagging.

Also, just confirming that data and documents are getting uploaded to dev.infrastructurelawguide.com. Please do not make any alterations to www.infrastructurelawguide.com.

Thanks,

Morgan

Hi Morgan ,

We are doing data upload using batch process and it is automated process and you have provided link for documents it seems like do manually open each link than take file name and we are seeing there are approximately 3000 around documents so can you please suggest or provide to take all documents name in order to complete data upload process.


Thanks,
Jitesh

Hi Jitesh ​,

As I mentioned above, all the documents have the UIN in the filename. My suggestion is that you use this to match the UIN in the master list. This should be easily done with Excel once you populate the filenames into the master list and run a few functions. Katrina ​ can assist with the process as required.

Thanks,

Morgan

Hi Morgan ,

once you populate the filenames into the master list

That is i am talking about, it will take long process to populate filesname into excel sheet from url. Can you please provide other way or from some where to take all filesname rather than to open url.


Thanks,
Jitesh

Hi Jitesh ​,

Can't you take the filenames from the applicable folders in Egnyte and then match them with the UIN?

Katrina ​, Can you please deal with this tomorrow.

Thanks,

Morgan

Hi Morgan and Jitesh ,

I will add in the filenames and upload a new version of the master list shortly. 

Thank you, 
Katrina

Hello Jitesh and Morgan ,

I have updated the master list to include a column with the pdf file names. Please let me know if you have any comments or questions. 

Thank you,
Katrina

ILG Project and Document Master List - UPLOAD V2.xlsx 7.27 MB • Download

Perfect. Thanks Katrina ​!

Morgan

Hi Morgan and Katrina ,

We have executed batch process and data uploaded on dev site (http://dev.infrastructurelawguide.com), Please check it and provide your feedback.

Thanks,
Jitesh

Hi Morgan and Katrina ,

Data upload has been done on dev site, Can you please tell me everything is fine? so we can do same process on live site (http://infrastructurelawguide.com/Admin)

Thanks,
Jitesh

Hi Jitesh ​,

I'll be take a closer look when I'm back from holidays next week.

Note we will not be updating the live site until the SME trial is complete.

Thanks,

Morgan

Hello Jitesh and Morgan ,

I've taken a look at the development site and had some trouble opening the original pdfs. Sometimes the documents will load but more often than not I receive the following error message:

dev site - pdf error.png 22.7 KB • Download

Thanks, 
Katrina

Hi Katrina ,

Could you report this as a to-do in the following group: ILG Testing Feedback - Fix Now - TOLOGIX - Infrastructure LawGuide (ILG) for Tologix and assign the task to Jitesh and Hiren . Also, mark it as a priority item in the to-do title.

Thanks,

Morgan

Hi Morgan , 

I have added this task to the the to-do list as a priority item. 

Thanks, 
Katrina

Great. Thanks Katrina .

I'll give you a call later today to catch up.

Thanks,

Morgan

Hi Morgan ,

As per discussion with Ketan , Katrina  has updated ILG files of manual conversion and make a green row in shared spreadsheet. like split big files into smaller one so can you please tell me where we will get those updated/divided ILG files.

https://docs.google.com/spreadsheets/d/1nI-SZFYaOAfSFU0eyrOmDHmNdFseppHOwC5DoRyR5Og/edit#gid=597349052

Actually we have already proceed ILG files those are poor quality and you have provided first slot of ILG files from ILG PDF quality list which you have provided earlier. your subdivided ( big files to smaller ) files and source of those files we are not aware, Can you please provide more information so we can correct and proceed files accordingly.


Thanks,
Jitesh

Hello Jitesh , 

Please find attached the revised version of the master list. 

Thanks,
Katrina

ILG Project and Document Master List - UPLOAD V3.xlsx 7.49 MB • Download

Hello everyone, 

Please find attached an updated version of the master list. I am re-uploading it in this thread so it is easily accessible for everyone. This master list has been revised to corrected the technical advisors issue and to include placeholder documents.

Thanks,
Katrina

ILG Project and Document Master List - UPLOAD V3.xlsx 12.9 MB • Download