TOLOGIX - ISLG App Rebuild

Problem with auto-IP session tracking in new application

Assigned to
Harsh Parikh, Tech Lead at DevIT Harsh P.
Notes
Further to video below, we have noticed some anomalous usage on certain accounts where auto-IP sessions are getting reported on group accounts where the sessions have no duration and no pages viewed. For example, see the spike in usage for the Allen & Overy LLP dozens of sessions were registered on April 26 at 16:16:
This makes me concerned that we have not applied the client URL validation process in the new application, similar what we did in the legacy application, which eliminated this problem with tracking auto-IP usage:
Could you please confirm whether the client URL validation process has been applied in the new ISLG application and whether this is the cause of the problem with Allen & Overy's usage details.

 

Comments & Events

Harsh Parikh, Tech Lead at DevIT
Hi Morgan Maguire, CEO Morgan ,

There is no need to validate client URL in new rebuild application as we are not matching with URL. if IP address is between the range then it will be automatically login.

Could you please check some other Client URL Validated account ? 
Morgan Maguire, CEO
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

Understood. Perhaps this isn't a client URL validation issues. I raised it as a possible cause for the anomalous usage we saw on the Allen & Overy LLP account. If it's not a client URL validation issue, could you please investigate further on what caused the anomalous usage on their account on 26 April 2021 at approximately 16:15 EST. We want to ensure we understand what happened here so that we can resolve and prevent it from happening again on this account and other accounts on the system. We had problems in the past accurately tracking auto-IP usage on the legacy application, and we want to avoid a similar problems here.

Thanks,

Morgan 
Harsh Parikh, Tech Lead at DevIT
Hi Morgan Maguire, CEO Morgan ,

The 26th April usage might be causing issue due to there was mistake in deployment. Please check that account on regular basis and if we still have an issue then will take remote call with Allen & Overy LLP account to find out the issue.

As, We checked on staging.islg and app.islg the auto-login session data are accurate with our IP ranges.

If possible then could you please investigate that which browser they are using for auto-login so we can check on that browser. We already checked in Chrome, FireFox and Edge and all browsers working fine.

 
Morgan Maguire, CEO
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

Looking through the usage details, the issue doesn't appear to be isolated to that date. Further to the screenshot below, there was a similar issue a few days ago on May 3rd:


I'll add this to the agenda for discussion tomorrow.

Thanks,

Morgan
Morgan Maguire, CEO
Hi Harsh Parikh, Tech Lead at DevIT Harsh and Martin Laporte, CTO at Tologix Martin ,

In preparation for discussing this issue tomorrow, a number of our larger enterprise subscribers use resource management tools like Research Monitor, Onelog and Lookup Precision.

In addition, many larger firms are using tools like Zscaler for security purposes and many academic institutions use products like EZproxy to remotely authenticate users.

Is it possible that these products might be causing the issues above, and if so, why are we not having similar problems in the legacy application?

Thanks,

Morgan
Martin Laporte, CTO at Tologix
Hi Morgan Maguire, CEO Morgan and Harsh Parikh, Tech Lead at DevIT Harsh ,

I will need to better understand how we operate behind the scene before being able to answer that.
In the meantime, I queried our database to see how many "zero second" sessions were logged since April 1st, and I counted over 2400 distinct sessions.
I am attaching the export of my query.


Harsh Parikh, Tech Lead at DevIT Harsh : can you help me understand what these "zero second" entries mean?

EDIT: note that my query only matched GroupName when a sessionMode was 'user/pass'. So if the sessionMode is Auto-IP, the GroupName will be NULL.
Morgan Maguire, CEO
Hi Harsh Parikh, Tech Lead at DevIT Harsh and Ketan Sondarva, Technical Project Manager at DevIT Ketan  

Please note that I moved up the priority on this to-do to #9 to ensure this gets addressed as the next item in the backlog.

Thanks,

Morgan
Harsh Parikh, Tech Lead at DevIT
Hi Martin Laporte, CTO at Tologix Martin ,

Could you please provide your query that how you export this data and which database you are using to execute the query. Also, There was issue for auto-login where end time was not updated if user close the browser tab or window which we resolved on app.islg by 27th April.
Martin Laporte, CTO at Tologix
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

I queried against ISLGRebuildProduction database, and the query I built is:
select s.LastName, s.FirstName, g.GroupName, ss.*
from dbo.SubscriberSession SS
left join dbo.Subscribers S on ss.SubscriberId = s.SubscriberId
left join dbo.Groups G on s.GroupId = g.GroupId
where ss.StartTime = ss.EndTime
and year(ss.StartTime) = 2021
order by ss.StartTime desc
Note that there still seems to be dozen of entries daily where start time and end time are equal.
Harsh Parikh, Tech Lead at DevIT
Hi Martin Laporte, CTO at Tologix Martin ,

Thanks. We checked on app.islg with auto-ip and credentials and its working fine to update session start time and end time. The end time is updating after every 5 second in to database.

Is this scenario that users logout or close the window before 5 seconds ? As it is hard to find what is the issue .

Could you please suggest Industrial team to do some testing on different browsers and check the session data?
Martin Laporte, CTO at Tologix
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

I dug a bit in the data, and realized that the % of "Zero seconds" sessions were very high in April, and even higher so far in May.
  • In April, 2409 out of 7359 (32.7%) were logged as "zero seconds" sessions.
  • In May so far, 2149 out of 4831 (44.5%) were logged as "zero seconds" sessions.
To be clear, if the difference in seconds between endTime and startTime is less than 1, I classify the session as "zero seconds".

I'm attaching an updated spreadsheet, which contains ALL session data for 2021.
Martin Laporte, CTO at Tologix
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

Also note that the vast majority of "zero second" sessions originate from Auto-IP. The highlighted row demonstrates that.
It seems to me we are still having an issue with the way sessions are handled for auto-IP users.
Harsh Parikh, Tech Lead at DevIT
Hi Martin Laporte, CTO at Tologix Martin ,

Is there any way to we ask to any group that which browser they are using for auto-login and Dose they use proxy server ?

As, When we are doing testing it seems working fine for both auto-ip and credentials login.
Martin Laporte, CTO at Tologix
Hi Morgan Maguire, CEO Morgan ,

I drilled down in the data a bit more this morning, and there is one IP address that jumps out in terms of # of "zero seconds" sessions:


That IP belongs to Allen & Overy LLP.
Out of a total combined 4831 sessions logged against ISLG, 1841 of them were "zero seconds" sessions from Allen & Overy LLP.

Furthermore, 1471 of these "zero seconds" sessions at Allen & Overy LLP occurred yesterday 5/12 between 9am and 9:30am (local server time). Since it's not humanely possible to create that many sessions in a span of 30 minutes, we are either dealing with a bug on our end, or with a bot or web crawler on their end.

I think we need to contact them and try to determine what is happening.

I also have a few questions:
  1. Can we tell how many distinct users from Allen & Overy LLP have accessed ISLG in May? All I can see in ISLG Admin are "Anonymous".
  2. Do we know whether this firm accesses ISLG via a proxy server or any other special access method?
  3. Have they reached out to us with any issue or concern since our new ISLG went live?

Thanks,
--Martin
Morgan Maguire, CEO
Ok. Thanks Martin Laporte, CTO at Tologix Martin . I believe Allen & Overy had some trouble accessing the system when we first launch the new application, because the new URLs had not been whitelisted in Zscaler, but we haven't heard about any issues since they resolved things on their end. I suspect they are using some kind of proxy server to give user auto-IP access, because all their users are working remotely. I'll forward you the email correspondence and we'll setup a meeting with them to resolve the issue.

Thanks,

Morgan
Martin Laporte, CTO at Tologix
Hi Morgan Maguire, CEO Morgan and Harsh Parikh, Tech Lead at DevIT Harsh ,

Note that the other top 2 IPs that have experienced the most zero-login issues are:
  • 154.59.124.193 (Not sure who this is – I will need to look them up in the database) | 289 total zero-seconds sessions
  • 195.33.27.60 (Wilmer Cutler Pickering Hale and Dorr LLP) | 158 total zero-seconds sessions

These 2 have generated far less zero-seconds sessions, but I am mentioning them just in case they may have reported issues lately.
Morgan Maguire, CEO
OK. Sounds good Martin Laporte, CTO at Tologix Martin . Looks like 154.59.124.193 is the Peace Palace Library.

Liam Murphy-Burke, Strategic Account Manager at Tologix Liam , could you reach out to both these subscribers and attempt to setup meetings with Martin Laporte, CTO at Tologix Martin ? Hopefully we can find a common thread between these three accounts and resolved the problems across all accounts.

Thanks,

Morgan
Liam Murphy-Burke, Strategic Account Manager at Tologix
Hi Morgan Maguire, CEO Morgan and Martin Laporte, CTO at Tologix Martin ,

I've sent those emails and CC'd both of you, Martin, I'm happy to join the calls if you'd like me to, but may be easier to navigate schedules with them yourselves (all 3 are European based).
Martin Laporte, CTO at Tologix
Thanks, Liam Murphy-Burke, Strategic Account Manager at Tologix Liam .

Harsh Parikh, Tech Lead at DevIT Harsh , I am focusing on the top 3 customers where zero-seconds sessions are logged and noticing some patterns.

One pattern is that when this issue occurs, several sessions will be created at the same time, but typically only one will actually be valid.
Example using Peace Palace Library:

Example using Allen & Overy:


The other pattern we need to explore is the actual date and time when these zero-seconds sessions occurs.
Both examples above (from 2 different customers) occurred at exactly 5:11am on May 11th.

I have not yet analyzed this pattern in-depth. I plan on continuing on that path tomorrow.
 
I am hoping the clues I provided will help you and the dev team troubleshoot further.
Martin Laporte, CTO at Tologix
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

The more I look at the data, the more I believe we have a bug.
Below is a chart showing the # of sessions that were less than 1 minute in duration AND had a PageViewCount of NULL.
You can see that we were fine for the first month after going live (most of April). The issue started happening more frequently on April 26.
Looking at the top 3 IP addresses with the issue, they all started showing this problem on or after April 26.

So the question is: what changed on or just before April 26 on the server?


Thanks,
--Martin
Harsh Parikh, Tech Lead at DevIT
Hi Martin Laporte, CTO at Tologix Martin ,

We have deployed one change on 26th April for auto-login that we resolved the issue that session time is end when user close the browser tab or windows which we deployed on 26th April.

The above mail is showing old legacy application URL. If Allen & Overy following any setting for www.islg.com URL then they should also follow same rule for our new ISLG domain URL. http://app.investorstatelawguide.com/

I am not sure but let's talk to them for this URL also.

I also instruct our QC to test different scenario with auto-login on staging.islg to catch this issue.
Martin Laporte, CTO at Tologix
Hi Harsh Parikh, Tech Lead at DevIT Harsh ,

We need to look at the code changes related to Auto-IP that were made and committed on April 26th.

Per the chart above, we were not encountering this issue before April 26th, and we now have several customers logging many "zero seconds sessions" since that date.

Since I do not have access to the source yet, we can either have a Zoom session where you show me what has changed, or if it's easier, you can also document here with the details.

Let me know what you prefer.

Thanks,
--Martin
Harsh Parikh, Tech Lead at DevIT
Hi Martin Laporte, CTO at Tologix Martin ,

Can we take call on Skype by tomorrow (20th May) 5:00 PM IST to discuss this thing ?
Martin Laporte, CTO at Tologix
Harsh, Darsh and myself met this morning to further discuss this topic and look at the session data.
Since we have not seen any type of spike in terms of "zero seconds" sessions since May 12th, we have decided to continue observing the situation and see if this will reoccur.

Additionally, talking with the IT team at Allen & Overy, I was able to confirm that one of their users sent thousands of requests to our server on May 12th. This seems to confirm that the issue is not caused by our code, as we truly received thousands of requests in the span of 30 minutes.
Allen & Overy uses Zscaler Proxy, which might have contributed to the problem. I have asked their IT group to add www.investorstatelawguide.com to their SSL Bypass category. This might end up resolving the issue.

I will continue monitoring the sessions for the next few weeks and report back in this thread.
Morgan Maguire, CEO
Ok. Thanks for the update Martin Laporte, CTO at Tologix Martin . Does this mean we should contact other subscribers experiencing a similar issue and request that they add www.investorstatelawguide.com to their SSL Bypass category?

Morgan
Martin Laporte, CTO at Tologix
Hi Morgan Maguire, CEO Morgan ,

I think we need more info before getting back to other subscribers. 

Allen & Overy account for 80% of all anomalous sessions, so my plan was to work with them first while monitoring all sessions for the next 1-2 weeks and see if (and where) the issue reoccurs.
This should give us more details and help us decide on next steps.
Morgan Maguire, CEO
Ok. Sounds good, Martin Laporte, CTO at Tologix Martin . Let us know when you have a more definitive understanding of the issues, and then we'll communicate with other clients accordingly.

Thanks,

Morgan
Martin Laporte, CTO at Tologix
Quick update: we have not had any spike in terms of anomalous sessions.
Below is the latest graph showing anomalous sessions.
I will look at the data in a week from now and report back.
Morgan Maguire, CEO
That's great to hear Martin Laporte, CTO at Tologix Martin .

As discussed, let's continue to monitor the situation over the next month, and assuming the issue doesn't occur again, we'll resolve the to-do.

Thanks,

Morgan
Martin Laporte, CTO at Tologix
We have not seen any spike in anomalous sessions this week.
Will continue to monitor throughout June and report weekly.

Morgan Maguire, CEO
Great. Thanks for the update Martin Laporte, CTO at Tologix Martin .

Morgan
Martin Laporte, CTO at Tologix
Outside of the small spike on June 8th (coming from Peace Palace), there were no other anomalous sessions to report for the past 10 days.

I have sent an email to Aad Janson at Peace Palace to inquire about the June 8th spike.

Morgan Maguire, CEO
OK. Thanks Martin Laporte, CTO at Tologix Martin . I guess we should continue to monitor for more anomalous usage events and deal with them on a subscriber by subscriber basis?
Martin Laporte, CTO at Tologix
Hi Morgan Maguire, CEO Morgan , yes I plan on monitoring weekly and addressing anomalous events on a subscriber by subscriber basis. 
Do you want me to continue reporting in this thread when I find anomalous sessions?
Morgan Maguire, CEO
Sounds good, Martin Laporte, CTO at Tologix Martin .

Yes, makes most sense to continue reporting things here.

Morgan
Paul Moon
Hi Martin Laporte, CTO at Tologix Martin :

It appears we haven't been following this recently. Is there a point of continuous reporting here or could we mark this complete?

Thanks,

Paul
Martin Laporte, CTO at Tologix
Hi Paul Moon Paul ,

We can mark this complete.
I did monitor for a while longer, and the data was accurate.
Furthermore, we have not heard from any customer on that topic, so we can assume it is resolved.

Thanks,
-Martin
Paul Moon 👍
Paul Moon
Paul Moon completed this to-do.