For incoming requests with facet query parameters, add robots metatag with noindex value

Patch #5 works very nicely for facets 3.0 as well. We find this quite important for SEO and helps Google to stop crawling pages that have facet queries in the URL. This is dispite facet links do already have nofollow on each link apparently this is not enough.

Added patch #5 to version 2.0.6 due to crawlers recursing into links used for the autocomplete widget.

We do want the page to be indexed ONCE, but bots are not smart, and they will just keep going.

Part of the problem is how the autocomplete builds on itself. With zero terms, the links look like this:

"urls": {
  "history": "/majors?f[0]=search:history",
  "culture": "/majors?f[0]=search:culture",
  ...
}

If I look for 'philosophy' then the autocomplete URLs look like this:

"urls": {
  "history": "/majors?f[0]=search:history&f[1]=search:philosophy",
  "culture": "/majors?f[0]=search:culture&f[1]=search:philosophy",
  ...
}

And the crawler will just keep going.

This behavior needs to be configurable.
We use facet blocks as menus and want the search engine to follow the links.

Looking for this feature in 3.x

We're getting hammered by bots hitting un cached pages with various combinations of facets that rarely get used.