The Trailing-Slash Bug That Stopped Google Indexing Our Site

For a stretch of this summer, this website would not index properly. Not "ranked badly" — pages simply were not making it into Google reliably, and the ones that did arrived slowly. The site was built, fast, schema-marked, on a decent host, and had a valid sitemap listing every URL. On paper there was nothing wrong with it.

The cause turned out to be one character. /about loaded correctly. /about/ — same URL with a trailing slash — returned a 500 Internal Server Error. Every clean URL on the site had a broken twin, and we had no idea, because nobody types a trailing slash by hand and no page on the site linked to one.

We fixed it on 2 September 2026. This is what it looked like, why it mattered far more than a broken URL normally would, and the one command that would have caught it in about four seconds.

The short version

On Apache, a rewrite rule that maps clean URLs onto PHP files can match its own output. A request for /about/ becomes about/.php, then about/.php.php, until Apache hits its ten-redirect limit and returns a 500. Because Google reduces crawl rate across a whole site when it sees persistent server errors, the damage is not the broken URL — it is the slower crawling of everything else. The fix is two rules: a 301 that strips the trailing slash when the path is not a real directory, and a condition on the mapping rule so it only fires when the target .php file actually exists.

The symptom: a sitemap Google was ignoring

The first signal was Search Console, and it was not a dramatic one. Over a three-month window the site recorded 95 queries, roughly 450 impressions and a single click. For a new site that is a bad number but not an obviously diagnostic one — it reads like "too new, keep going", which is exactly how we read it for too long.

What did not fit was the coverage. URLs sat in Discovered – currently not indexed for weeks. Google knew the pages existed, had them from the sitemap, and was choosing not to fetch them. That is not the signature of weak content. Weak content gets indexed and then ignored. This was something upstream of content.

Finding it: check every URL, not the ones you link to

The mistake we had been making was testing the site the way a visitor uses it — clicking through from the home page. Every internal link on the site is generated, so every internal link was in the one shape that worked. To find this you have to request the shapes nobody on your site links to, because those are exactly the shapes an external link, an old bookmark or a mistyped URL will arrive in.

So we took every URL from the sitemap and requested each one twice, once as-is and once with a trailing slash:

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+' | cut -c6- \
  | while read -r u; do
      printf '%s  %s  %s\n' \
        "$(curl -s -o /dev/null -w '%{http_code}' "$u")" \
        "$(curl -s -o /dev/null -w '%{http_code}' "$u/")" \
        "$u"
    done

The output was unambiguous. Two columns of status codes, and the second column was 500 on every single line:

200  500  https://sitedigicraft.com/about
200  500  https://sitedigicraft.com/contact
200  500  https://sitedigicraft.com/services/seo
200  500  https://sitedigicraft.com/blog/why-website-speed-matters-more-than-you-think
...

Every clean URL on the site had a 500-error duplicate. The sitemap had 138 URLs in it. So did the shadow set.

The cause: a rewrite rule that matched its own output

This site runs on plain PHP with clean URLs, which means Apache has a rule that maps /about onto the file about.php without changing the address bar. The rule is the ordinary one that every "pretty URLs in PHP" tutorial gives you, and in its naive form it looks something like this:

RewriteRule ^(.*)$ $1.php [L]

Read that as a machine rather than as a human. A request for /about captures about and rewrites to about.php. Fine. But a request for /about/ captures about/ and rewrites to about/.php — a file that does not exist. The [L] flag does not mean "stop"; it means "stop this pass". Apache starts the rules again with the new path, which still matches ^(.*)$, and produces about/.php.php. Then about/.php.php.php.

Apache has a guard for this, LimitInternalRecursion, set to 10 by default. Once the tenth internal redirect goes by, it gives up and returns a 500, with a line in the error log that names the problem exactly:

Request exceeded the limit of 10 internal redirects
due to probable configuration error.

That log line is the whole diagnosis in one sentence. We had never read it, because nothing on the site was generating a request that triggered it.

Why a 500 is so much worse than a 404 here

This is the part that makes a trailing-slash bug more than a cosmetic problem, and it is the part most write-ups about rewrite loops skip.

Googlebot treats the two error classes completely differently, and sensibly so:

Response	What Google concludes	Effect on crawling
404 / 410	This page does not exist	Drops that URL. Rest of the site unaffected.
500 / 503	This server is in trouble	Backs off crawling the whole site to avoid making it worse.

A 404 is information. A 500 is a request to go away, and Google's own crawling documentation is explicit that persistent server errors cause it to reduce crawl rate. That is the correct behaviour — if a server is falling over, hammering it harder is the last thing a well-behaved crawler should do.

So the damage was never really about the broken URLs themselves. Nobody was visiting /about/. The damage was that from Googlebot's point of view this was a site where a meaningful share of requests returned server errors, so the sensible thing to do was crawl it slowly and carefully. Which it did, for weeks, while we wrote more content and wondered why none of it was landing.

It also explains the shape of that Search Console data. A three-month window that mostly sits inside a period when the crawler was deliberately throttling itself is not a measurement of how good the content is. It is a measurement of a configuration error.

The fix: two rules, and the order matters

There are two separate jobs. The first is to decide what a trailing slash means and redirect accordingly. The second is to stop the mapping rule from ever being able to match its own output again, so the same class of bug cannot come back through a different door.

The redirect, which belongs near the top of the file:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.+)/$ /$1 [R=301,L]

The !-d condition is not optional. Without it this rule also strips the slash from real directories, and then Apache's own DirectorySlash puts it back, and you have built a redirect loop while fixing one. The condition says: only strip the trailing slash if this is not an actual directory on disk.

Then the mapping rule itself, hardened:

RewriteCond %{REQUEST_FILENAME} !-d
RewriteCond %{REQUEST_FILENAME}.php -f
RewriteRule ^(.*[^/])$ $1.php [L]

Two independent defences, either of which would have prevented the original loop:

• %{REQUEST_FILENAME}.php -f — only rewrite if the target file actually exists. about/.php does not exist, so the rule never fires, so there is nothing to loop on.
• ^(.*[^/])$ — the pattern now refuses to match anything ending in a slash in the first place.

Belt and braces on purpose. A guard that only works because of a condition two lines above it is the kind of thing that breaks the next time someone edits the file.

One ordering detail worth knowing if you have short marketing URLs. We send /review to the review form with a fragment, and that rule has to sit above the trailing-slash rule, or /review/ costs two redirects instead of one. It also needs the NE flag, or Apache helpfully escapes the # to %23 and the fragment stops working:

RewriteRule ^(review|write-review)/?$ /reviews#write-review [R=301,L,NE]

Check your own site in one command

If you run clean URLs on Apache — plain PHP, a hand-rolled router, or a framework someone configured for you years ago — it is worth four seconds of your time:

curl -s -o /dev/null -w '%{http_code}\n' https://yoursite.com/about/

What you want back is 301. 200 is acceptable but means you are serving the same content at two URLs, so check that the canonical tag points at one of them. 500 means you have the bug described here. 404 is untidy but harmless.

Then try the variants that external links actually arrive in, because each one is a different code path: /About with a capital letter, /about.php directly, /about//, and //about. We found a second, smaller problem the same way — direct .php URLs needed a 301 to the clean version, but never on a POST request, because a redirect converts the POST to a GET and the browser drops the form body. That one was silently eating review submissions, and the symptom was a form that appeared to work and saved nothing.

What we would do differently

Honestly: watch Search Console's crawl stats before its performance report. The performance numbers tell you how you are doing in a competition. The crawl stats tell you whether you have been allowed to enter. We spent weeks optimising content for a crawler that had decided our server was unwell.

The other lesson is cheaper. A sweep of every URL in your sitemap, in a few shapes, belongs in whatever you run before a deploy. It is a dozen lines of shell, and it runs in seconds. We now check the whole URL list after every change to this site rather than trusting that the templates are still producing what they produced last month.

We should also be honest about the limits of this story. We cannot separate how much of the site's weak early performance was this bug and how much was simply being a new site in a competitive market — both were true at the same time, and there is no clean experiment available after the fact. What we can say is narrower: the error was real, it affected every URL, it persisted for weeks, Google documents that persistent 5xx responses reduce its crawl rate, and the trailing-slash requests have returned a single 301 since the fix.

If this is the sort of thing you would rather have somebody else find on your site, it is part of what our SEO work and website care plans cover. And if your site is misbehaving in a way you cannot pin down, tell us what it is doing — the sweep above is the first thing we run, and you can run it yourself from this page without involving us at all.

Related reading: why your website is not showing up on Google, which works through the other causes in the order they are worth checking, and the on-page SEO checklist for what to do once crawling is healthy again.