✔ Problem with auto-IP session tracking in new application
Completed by Paul M.
- Assigned to
-
Harsh P.
- Notes
-
Further to video below, we have noticed some anomalous usage on certain accounts where auto-IP sessions are getting reported on group accounts where the sessions have no duration and no pages viewed. For example, see the spike in usage for the Allen & Overy LLP dozens of sessions were registered on April 26 at 16:16:
This makes me concerned that we have not applied the client URL validation process in the new application, similar what we did in the legacy application, which eliminated this problem with tracking auto-IP usage:
Could you please confirm whether the client URL validation process has been applied in the new ISLG application and whether this is the cause of the problem with Allen & Overy's usage details.
There is no need to validate client URL in new rebuild application as we are not matching with URL. if IP address is between the range then it will be automatically login.
Could you please check some other Client URL Validated account ?
Understood. Perhaps this isn't a client URL validation issues. I raised it as a possible cause for the anomalous usage we saw on the Allen & Overy LLP account. If it's not a client URL validation issue, could you please investigate further on what caused the anomalous usage on their account on 26 April 2021 at approximately 16:15 EST. We want to ensure we understand what happened here so that we can resolve and prevent it from happening again on this account and other accounts on the system. We had problems in the past accurately tracking auto-IP usage on the legacy application, and we want to avoid a similar problems here.
Thanks,
Morgan
The 26th April usage might be causing issue due to there was mistake in deployment. Please check that account on regular basis and if we still have an issue then will take remote call with Allen & Overy LLP account to find out the issue.
As, We checked on staging.islg and app.islg the auto-login session data are accurate with our IP ranges.
If possible then could you please investigate that which browser they are using for auto-login so we can check on that browser. We already checked in Chrome, FireFox and Edge and all browsers working fine.
Looking through the usage details, the issue doesn't appear to be isolated to that date. Further to the screenshot below, there was a similar issue a few days ago on May 3rd:
I'll add this to the agenda for discussion tomorrow.
Thanks,
Morgan
In preparation for discussing this issue tomorrow, a number of our larger enterprise subscribers use resource management tools like Research Monitor, Onelog and Lookup Precision.
In addition, many larger firms are using tools like Zscaler for security purposes and many academic institutions use products like EZproxy to remotely authenticate users.
Is it possible that these products might be causing the issues above, and if so, why are we not having similar problems in the legacy application?
Thanks,
Morgan
I will need to better understand how we operate behind the scene before being able to answer that.
In the meantime, I queried our database to see how many "zero second" sessions were logged since April 1st, and I counted over 2400 distinct sessions.
I am attaching the export of my query.
EDIT: note that my query only matched GroupName when a sessionMode was 'user/pass'. So if the sessionMode is Auto-IP, the GroupName will be NULL.
Please note that I moved up the priority on this to-do to #9 to ensure this gets addressed as the next item in the backlog.
Thanks,
Morgan
Could you please provide your query that how you export this data and which database you are using to execute the query. Also, There was issue for auto-login where end time was not updated if user close the browser tab or window which we resolved on app.islg by 27th April.
I queried against ISLGRebuildProduction database, and the query I built is:
Thanks. We checked on app.islg with auto-ip and credentials and its working fine to update session start time and end time. The end time is updating after every 5 second in to database.
Is this scenario that users logout or close the window before 5 seconds ? As it is hard to find what is the issue .
Could you please suggest Industrial team to do some testing on different browsers and check the session data?
I dug a bit in the data, and realized that the % of "Zero seconds" sessions were very high in April, and even higher so far in May.
I'm attaching an updated spreadsheet, which contains ALL session data for 2021.
Also note that the vast majority of "zero second" sessions originate from Auto-IP. The highlighted row demonstrates that.
It seems to me we are still having an issue with the way sessions are handled for auto-IP users.
Is there any way to we ask to any group that which browser they are using for auto-login and Dose they use proxy server ?
As, When we are doing testing it seems working fine for both auto-ip and credentials login.
I drilled down in the data a bit more this morning, and there is one IP address that jumps out in terms of # of "zero seconds" sessions:
That IP belongs to Allen & Overy LLP.
Out of a total combined 4831 sessions logged against ISLG, 1841 of them were "zero seconds" sessions from Allen & Overy LLP.
Furthermore, 1471 of these "zero seconds" sessions at Allen & Overy LLP occurred yesterday 5/12 between 9am and 9:30am (local server time). Since it's not humanely possible to create that many sessions in a span of 30 minutes, we are either dealing with a bug on our end, or with a bot or web crawler on their end.
I think we need to contact them and try to determine what is happening.
I also have a few questions:
--Martin
Thanks,
Morgan
Note that the other top 2 IPs that have experienced the most zero-login issues are:
Thanks,
Morgan
I've sent those emails and CC'd both of you, Martin, I'm happy to join the calls if you'd like me to, but may be easier to navigate schedules with them yourselves (all 3 are European based).
One pattern is that when this issue occurs, several sessions will be created at the same time, but typically only one will actually be valid.
Example using Peace Palace Library:
Example using Allen & Overy:
The other pattern we need to explore is the actual date and time when these zero-seconds sessions occurs.
Both examples above (from 2 different customers) occurred at exactly 5:11am on May 11th.
I have not yet analyzed this pattern in-depth. I plan on continuing on that path tomorrow.
I am hoping the clues I provided will help you and the dev team troubleshoot further.
The more I look at the data, the more I believe we have a bug.
Below is a chart showing the # of sessions that were less than 1 minute in duration AND had a PageViewCount of NULL.
You can see that we were fine for the first month after going live (most of April). The issue started happening more frequently on April 26.
Looking at the top 3 IP addresses with the issue, they all started showing this problem on or after April 26.
So the question is: what changed on or just before April 26 on the server?
Thanks,
--Martin
See attached reply from Allen & Overy and let us know your thoughts.
Thanks,
--Martin
We have deployed one change on 26th April for auto-login that we resolved the issue that session time is end when user close the browser tab or windows which we deployed on 26th April.
The above mail is showing old legacy application URL. If Allen & Overy following any setting for www.islg.com URL then they should also follow same rule for our new ISLG domain URL. http://app.investorstatelawguide.com/
I am not sure but let's talk to them for this URL also.
I also instruct our QC to test different scenario with auto-login on staging.islg to catch this issue.
We need to look at the code changes related to Auto-IP that were made and committed on April 26th.
Per the chart above, we were not encountering this issue before April 26th, and we now have several customers logging many "zero seconds sessions" since that date.
Since I do not have access to the source yet, we can either have a Zoom session where you show me what has changed, or if it's easier, you can also document here with the details.
Let me know what you prefer.
Thanks,
--Martin
Can we take call on Skype by tomorrow (20th May) 5:00 PM IST to discuss this thing ?
Since we have not seen any type of spike in terms of "zero seconds" sessions since May 12th, we have decided to continue observing the situation and see if this will reoccur.
Additionally, talking with the IT team at Allen & Overy, I was able to confirm that one of their users sent thousands of requests to our server on May 12th. This seems to confirm that the issue is not caused by our code, as we truly received thousands of requests in the span of 30 minutes.
Allen & Overy uses Zscaler Proxy, which might have contributed to the problem. I have asked their IT group to add www.investorstatelawguide.com to their SSL Bypass category. This might end up resolving the issue.
I will continue monitoring the sessions for the next few weeks and report back in this thread.
Morgan
I think we need more info before getting back to other subscribers.
Allen & Overy account for 80% of all anomalous sessions, so my plan was to work with them first while monitoring all sessions for the next 1-2 weeks and see if (and where) the issue reoccurs.
This should give us more details and help us decide on next steps.
Thanks,
Morgan
Below is the latest graph showing anomalous sessions.
I will look at the data in a week from now and report back.
As discussed, let's continue to monitor the situation over the next month, and assuming the issue doesn't occur again, we'll resolve the to-do.
Thanks,
Morgan
Will continue to monitor throughout June and report weekly.
Morgan
I have sent an email to Aad Janson at Peace Palace to inquire about the June 8th spike.
Do you want me to continue reporting in this thread when I find anomalous sessions?
Yes, makes most sense to continue reporting things here.
Morgan
It appears we haven't been following this recently. Is there a point of continuous reporting here or could we mark this complete?
Thanks,
Paul
We can mark this complete.
I did monitor for a while longer, and the data was accurate.
Furthermore, we have not heard from any customer on that topic, so we can assume it is resolved.
Thanks,
-Martin