✔ New site not indexing on major search engines
Completed by Morgan M.
- Assigned to
-
Anil V.
Harsh P.
Ryan K.
- Due on
- Notes
-
Further to screenshot below, I've noticed that the sub-pages displayed on Google and Bing are still from our old site. Is there something we're doing on our back-end that is preventing these search engines from indexing the current site?
- Look for indexing errors, and submit for re-index
Thanks,
Morgan
I've set the site up in Google Webmaster Tools and am awaiting a scan for me to see indexing data.
I've also registered for a Bing Webmaster Tools account.
Thanks!
Ryan
We have uploaded above XML file in to root of the www.islg. Please verify it.
In addition to the issue mentioned in my last comment, please add the following to the <head> of the Admin page at https://www.investorstatelawguide.com/admin/adminlogin
We need to do above change directly on www.islg. Hence, Next Monday we will do above change.
Hope this is fine.
Morgan
We have set above meta tag inside the <head> on www.islg.
If any user will try to access the unpublished page or try to access the page which is not available in ISLG then application will automatically throw out the user on following URL.
http://dev.investorstatelawguide.com/Error/SubscriberPageNotFound
Please check and let us know the feedback.
For archived web pages, it seems like they're not redirecting to a 404 page?
See "The Relevance of GATT 1947 and GATT Code Jurisprudence" https://www.investorstatelawguide.com/nonMember/nonMember?Id=32 as an example
Currently, We have set up 404 page on dev.islg. We haven't migrated on www.islg. If you check above URL on dev.islg then it will redirect to proper 404 page. Hence, Please check on dev.islg and all will be OK then we will migrate this on next Monday.
Only update is to please change "Try click the button..." to "Click the button..."
Thanks!
Ryan
I've done some further testing and it appears as though we're just 302 redirecting the archived page (e.g. http://dev.investorstatelawguide.com/nonMember/nonMember?Id=93) to a page with 404 content on it. The issue is that the archived page isn't returning a 404 response code which will tell Google to remove it from the index.
I've chatted with Juan and he's sent me some more info https://www.bruceclay.com/blog/microsoft-iis-custom-404-error-page-configuration/
It means, We should have to redirect the archived page on 302 page with same content as 404 ? and other pages will redirect to 404 Page.
Is it right ??
There is no need for a redirect when you are handling 404 errors.
The server should respond immediately with a 404 when the page or resource does not exist.
In a nutshell, if the user requests a url like say:
http://www.yourwebsite.com/path/non-existing-pagex
The server should return 404 HTTP status code in the header of the response. In the body of the response you can display a message explaining to the user that the page does not exist. In your case this should also be the case for archived/unpublished pages.
If you are working with classic asp, you would use the Response object to do that like:
Response.Status = "404 File Not Found"
....Then the body of the page.
This ensures that both search engines and users understand that the requested url does not exist.
I suppose the redirect may achieve the same end result, but it changes the url to something else (your generic 404 actual page). You are also making it worse by returning a 200 status code on that 404 template page. If you leave as is, the urls indexed by google will keep appearing in the search results because google won't know that your 404 template page is meant to be a missing resource.
Hope this clarifies things.
Please check and confirm.
Could you please check the 404 issue on dev.islg ? If all will be OK then will migrate this task on Monday.
I'm just awaiting talking to Juan, but I think we'll probably have some more work to do on this. The 404 page is now returning a 404 response code, but the original non-existent/archived page is still redirecting to it, rather than returning its own 404 response code.
As an example, when I visit a non-existent page on our website (https://industrialagency.ca/doesntexist) the URL doesn't change, but we do get the Page not found content, with a 404 response code.
Any update on getting this resolved? Note that a few users have complained to us for not being able to login to the site after inadvertently attempting to login on the admin site. How long do you expect it will take for Google and Bing to remove the admin login page from it's list of indexed pages?
I also noticed that other odd pages are getting indexed (see screenshot)
The pages I would expect list are the following:
Login to Notepad: https://www.investorstatelawguide.com/Home/Login?Logout=Yes&autologin=n
Login: https://www.investorstatelawguide.com/User/Welcome
Product: https://www.investorstatelawguide.com/nonMember/nonMember?Id=265
Resources & Events: https://www.investorstatelawguide.com/nonMember/nonMember?Id=267
About ISLG: https://www.investorstatelawguide.com/nonMember/nonMember?Id=266
Contact Us: https://www.investorstatelawguide.com/nonMember/contactus
Also, is it possible to alter the URLs to the following (with the appropriate redirects from the old URLs):
Login to Notepad: https://www.investorstatelawguide.com/nonMember/NotepadLogin
Product: https://www.investorstatelawguide.com/nonMember/Product
Resources & Events: https://www.investorstatelawguide.com/nonMember/ResourcesEvents
About ISLG: https://www.investorstatelawguide.com/nonMember/AboutISLG
Contact Us: https://www.investorstatelawguide.com/nonMember/ContactUs
Start Your Trial: https://www.investorstatelawguide.com/nonMember/StartYourTrial
Thanks,
Morgan
I've gone through the search results, and manually submitted several URLs for removal from the index/cache at both Google and Bing. These requests will remove them temporarily (Bing says for 90 days). After that time they will re-scan for those pages, and we'll have to have the 404 issue sorted by then.
I've asked Juan to take another look at how we're handling 404s on dev.islg as I think there's still some work to be done there.
Ryan
Please review my previous post carefully. I did not suggest to use redirects for 404 errors, on the contrary I advised against it.
Here is the general suggested approach when you are migrating from a legacy web app or website to a new one where the url structure is changing:
For each old url:
- If the URL is for a resource that does exist in the new system, then return a 404 error for the request (no redirects).
https://en.wikipedia.org/wiki/HTTP_404
- If the URL is key for accessing the app (i.e. login urls) and it has been moved to a new URL, then return a 301 response (Moved Permanently). https://en.wikipedia.org/wiki/HTTP_301
Please let me know if this now clear. These will ensure that key urls are handled as 301 redirects for users who find them in search engine results. It will also indicate to the search engine that their index should be updated with the new location.
Also, this approach will ensure that resources that no longer exist are removed from the index.
Thanks for clarification. I understood and we are working on this.
Currently, We are facing an issue in IIS server configuration to call 404 page (with response code 404) from server without changing the URL.
We are looking into this and will update you soon.
Above feature for 404 is implemented on dev.islg. Please check and let us know the feedback.
Can you let us know if the updates Morgan requested at Re: New site not indexing on major search engines - TOLOGIX - ISLG Maintenance are possible?
Thanks!
Ryan
We have discussed this with our SEO team. As per their suggestion, we can remove all unnecessary URLs from Google webmaster tool (below URL).
https://www.google.com/webmasters/
For "Login to Notepad" : Go to "Fetch as Google" and index the URL for "Login to Notepad".
Also, is it possible to alter the URLs to the following (with the appropriate redirects from the old URLs):
I'm wondering, however, if we take this a step further and make some good, SEO-friendly URLs if we're going to do this work.
Login to Notepad: https://www.investorstatelawguide.com/notepad-login
Product: https://www.investorstatelawguide.com/product
Resources & Events: https://www.investorstatelawguide.com/resources-events
About ISLG: https://www.investorstatelawguide.com/about-islg
Contact Us: https://www.investorstatelawguide.com/contact-us
Start Your Trial: https://www.investorstatelawguide.com/start-your-trial
Lets get this implemented asap. Note that we'll need to ensure there are redirects for the old URLs because of third party links to the site.
Thanks,
Morgan
Thanks!
Ryan
I've taken a look at Bing again, and those URLs are on the list of blocked I added a week ago. They should be removed like Google has, but I found a blog post saying that Bing is a bit more finicky than Google. I'll continue to monitor. The sooner we can get the 404 updates on www.islg the better.
Ryan
Morgan
Yes, this redirection is very much possible. We will start development on it after completing bookmark task.
Could we migrate 404 functionality on www.islg to next Monday ?
Morgan
We are making only following URLs to SEO-friendly URLs.
Login to Notepad: https://www.investorstatelawguide.com/notepad-login
Product: https://www.investorstatelawguide.com/product
Resources & Events: https://www.investorstatelawguide.com/resources-events
About ISLG: https://www.investorstatelawguide.com/about-islg
Contact Us: https://www.investorstatelawguide.com/contact-us
Start Your Trial: https://www.investorstatelawguide.com/start-your-trial
Hope this is fine.
This task has also done in our local environment and will migrate this change on Monday with bookmark task.
As
We have migrated 404 page on www.islg. Please check and confirm.
Thanks,
Morgan
Further to my comment above, with
Thanks,
Morgan
One issue I noticed on the new URLs, the URLs for Notepad Login is: http://dev.investorstatelawguide.com/notepad-login?Logout=Yes&autologin=n rather than http://dev.investorstatelawguide.com/notepad-login. Is it possible to get this updated?
Thanks,
Morgan
Noted - I will see if someone on our team can take a look.
Thanks,
Stephen
Morgan
We have updated the Notepad Login URL as you suggested on dev.islg.
http://dev.investorstatelawguide.com/notepad-login
Please check and confirm.
Our team took a look and don't see any issues - that said I would suggest having
Thanks,
Stephen
Morgan
The new URLs look good, but the redirects from the old to the new URLs didn't work properly for the following:
Product (error): http://dev.investorstatelawguide.com/nonMember/nonMember?Id=265
Resources & Events (error): http://dev.investorstatelawguide.com/nonMember/nonMember?Id=267
About (error): http://dev.investorstatelawguide.com/nonMember/nonMember?Id=266
Contact Us (doesn't redirect to new URL): http://dev.investorstatelawguide.com/nonMember/contactus
This page should return the 404 page I believe, but it's just a text message. Could you please investigate?
https://www.investorstatelawguide.com/nonMember/nonMember?Id=32
Ryan
We have set proper 404 page for following URL.
https://www.investorstatelawguide.com/nonMember/nonMember?Id=32
Please check and confirm.
For Old URL to New URL, it means if any user browse the old URL (e.x http://dev.investorstatelawguide.com/nonMember/nonMember?Id=265) then it will be automatically redirect to New URL ?? or shows the 404 Page ?
Please clarify.
Morgan
We would still maintain the 404s for the archived/deleted/unpublished pages we worked on before.
Ryan
We are redirecting Old Urls to New Urls on dev.islg. Also,maintained the 404s for the archived/deleted/unpublished page.
Please check and confirm.
Looks good on dev.islg.
Ryan
Morgan
This task has been migrated on www.islg. Please check and confirm.
Morgan
Ryan