Angular2 App: Fetch as Google doesn't load page content

Question

I am working on Angular2 based web app. I used Angular CLI to generate app and then to build it for prod. I have hosted website on AWS S3 & Cloudfront. When I use 'Fetch as Google' tool from the webmaster, it shows only Loading....

Isn't Googlebot able to crawl my website?

Is your website frontend-only? If you don't have a backend to serve the file you could encounter this issue. — Syntactic Fructose, Apr 02 '17 at 05:59
I have used Angular2 for front-end and is served via AWS cloudfront. Back-end is nodejs and hosted on different sub-domain. Angular2 app makes API requests to nodejs app to render content. — Nilesh G, Apr 02 '17 at 06:02
Is your frontend being served by any backend? or is it a bare index.html sitting on your domain? — Syntactic Fructose, Apr 02 '17 at 06:04
No, the frontend is not served by any server side language. It is hosted on AWS S3 and served by Cloudfront. — Nilesh G, Apr 02 '17 at 06:13
Cross posted: http://webmasters.stackexchange.com/questions/105098/angular2-fetch-as-google-doesnt-render-page — Stephen Ostermiller, Apr 02 '17 at 10:27

score 18 · Accepted Answer · edited Dec 28 '18 at 14:34

18

had a similar issue. I believe Google-Bot do not support modern JS. I simply activated all shims recommended by angular.io see https://angular.io/docs/ts/latest/guide/browser-support.html and added in the script header:

<script src="https://cdnjs.cloudflare.com/ajax/libs/core-js/2.4.1/shim.min.js">
</script>

If you really need all of them I'm not sure, but I need support for older IE anyway.

If this works for all Searchbots is very unclear, if you have to support them you can try https://prerender.io/. However they are also just able to render it with the shims

Hope this helps.

edited Dec 28 '18 at 14:34

Stefan Falk

23,898
50
191
378

answered Jun 07 '17 at 20:47

Krosan

684
1
6
15

2

This ended up being the resolution to my problem. I can't thank you enough for sharing this solution. I used the cdn link you shared, but I'm curious how you "activated all shims recommended by angular.io". Can you explain that a little bit? – Engineer_Andrew Jul 17 '17 at 03:26
1

So i guess this means the cdn is sufficent to resolve the issue? Well in your angular.io project there should be a file called "polyfills.ts" where all the shims from https://angular.io/docs/ts/latest/guide/browser-support.html are commented out. I basically removed those comments for all classes which had something to do with IE. Which would be all the core-js/es6/*, the web-animations-js and classlist.js. Hopes this helps – Krosan Jul 18 '17 at 09:13
I've been having the same problem with googlebot not being able to render my angular 4 site using "fetch as google". It returned blank no matter what I tried. The solution here with including shim, was what finally solved the problem for me. Consider making this the accepted answer. – brinch Sep 01 '17 at 11:49
great answer. does the google bot also read the content of the site for SEO? – Andreas Feb 01 '18 at 10:34
1

@Andreas The bot is at least able to read the meta description field. If it can also follow links/routes I'm not sure since i just tested it on a small test project here http://modern-sub.net/ if you try to google "modern sub.net" you will find the same meta-description field. However i would assume that links also work since in the HTML it is a normal – Krosan Feb 02 '18 at 10:56
since many ppl. did see that post. At ng-conf 2018 there was a talk about SEO where the give more details: https://www.youtube.com/watch?v=fiT5g9KSxmw – Krosan May 07 '18 at 13:17
shouldn't webpack be transpiling to ES5 when it does the same for trypescript? Why should ES6 compatibility be an issue? – SeanMC Sep 12 '18 at 13:29
I never claimed that it is an ES6 compatibility issue. In fact i can not completly say what the missing piece is. – Krosan Sep 13 '18 at 14:26
Okay, stop right there. Yesterday I spent a whole day removing my app from my Java deployment and made it a standalone server side rendered application in order to get indexed by google. Now I just added this line and googlebot is also able to see my website. Please somebody explain to my why this works and what exactly I am doing to my application if I add this `shim.min.js` thing. – Stefan Falk Dec 22 '18 at 12:22
1

@StefanFalk sorry for you that you waisted so much time. It is a little bit a shame that there is no clear information existining about that. For the question about the shims. A shim just adds functionality to the JS browser functions. For example if the browser has no implementation for the string.repeat function it would just provide an own implementation on String.prototype.repeat to support "modern" code on the old browser. What they exactly changing can be seen on there sides: https://www.npmjs.com/package/core-js. FYI: i highly doubt that you need the CDN and polyfill – Krosan Dec 28 '18 at 13:37
1

Cool. Thanks for that additional info. By now I think I get what actually happened there. It would seem that Angular just uses new features which googlebot does not know about but loading `shim.min.js` corrects that for the reasons you stated. It really *is* a shame that there is so little information about this especially since Google should be interested in promoting Angular on one hand and making it accessible for googlebot on the other. For now I'll go that way and have a close look on what googlebot sees. The question is now: Why would anybody use Angular Universal for this instead? – Stefan Falk Dec 28 '18 at 14:33
as " Naveed Ahmed" pointed out in the last answer. Universal doesn't fix this. But universal is still very useful since it reduces loading time and loading times influences your SEO rank.Further low end devices like older Phones might benefit from the better performance. – Krosan Dec 28 '18 at 14:37
Thanks for the solution! This answer probably saved me hours/days worth of researching.. – burnt1ce Apr 20 '19 at 15:40

score 2 · Answer 2 · edited Dec 28 '18 at 14:33

First this was talked about at NG-Conf 2018 Apr For the slides click here

Looking at the source code of the Angular.io here is how the angular guys do this according to the slides

 <script>
    if (window.document.documentMode) {
      // polyfill IE11 in a blocking way
      var s = document.createElement('script');
      s.src = 'generated/ie-polyfills.min.js';
      document.head.appendChild(s);
    } else if (!Object.assign) {
      // polyfill other non-evergreen browsers in a blocking way
      var polyfillUrl = "https://cdn.polyfill.io/v2/polyfill.min.js?features=default,Array.prototype.find&flags=gated&unknown=polyfill";
      // send a blocking XHR to fetch the polyfill
      // then append it to the document so that its eval-ed synchronously
      // this is required because the method used for IE is not reliable with other non-evergreen browsers
      var xhr = new XMLHttpRequest();
      xhr.addEventListener("load", function() {
        var s = document.createElement('script');
        s.type = 'text/javascript';
        var code = this.responseText;
        s.appendChild(document.createTextNode(code));
        document.head.appendChild(s);
      });
      xhr.open("GET", polyfillUrl, false);
      xhr.send();
    }
  </script>

Add the above script in the HEAD section of your index file.

Worth to mention that if you go with the answer of just adding a CDN you most probably are loading a script that is not needed for most of the modern browsers to those browsers as well and should be avoided.

score 1 · Answer 3 · answered Apr 02 '17 at 06:21

1

Since your frontend isn't being served by any server side language, I'd recommend using Angular2-Universal to serve a static HTML site on initial load.

You can checkout their quickstart guide and get it working pretty fast.

answered Apr 02 '17 at 06:21

Syntactic Fructose

18,936
23
91
177

3

Isn't Googlebot supposed to render what a normal user sees on the browser? – Nilesh G Apr 02 '17 at 06:56
@NileshG Yes, and a normal users sees exactly that, an initial loading screen. – Syntactic Fructose Apr 02 '17 at 17:42
yea i would prefer an universal page. However it is simply not true that it is just working with that. see my answer above. – Krosan Feb 02 '18 at 10:57
3

Even with Universal you need to add the shims mentioned by @Krosan in the answer. – Naveed Ahmed Apr 16 '18 at 20:49

Angular2 App: Fetch as Google doesn't load page content

3 Answers3

Linked