Site Not Indexed in Bing: The Next.js and IndexNow Fix

Why does Google index my Next.js site while Bing shows nothing?

Because Google repairs your mistakes and Bing does not. Googlebot will follow a canonical that redirects, work out where you meant to point it, and index the destination anyway. Bing reads the same thing as a contradiction: the page says its real address is over there, and over there says the real address is back here. So the URL is dropped rather than resolved. In almost every case I have looked at, nothing is wrong with the content. The addressing is wrong.

I found this on my own domain. Every canonical tag, every hreflang alternate and all 96 URL occurrences in sitemap.xml named one host, while production was served from the other — and the host in the markup answered a 308 for every path, including /robots.txt and /sitemap.xml. Google had the site indexed. Bing Webmaster Tools was not indexing it, because I had handed Bing a sitemap in which not one URL returned 200.

The cost is no longer Bing traffic alone. DuckDuckGo's ten blue links come from Bing's index, Copilot and Windows search sit directly on it, and ChatGPT's live search leans on it. A site that is invisible in Bing is simultaneously invisible on the surfaces downstream of it, which is why I treat Bing crawlability as a prerequisite for AI crawler and GEO readiness rather than a separate job.

Is your canonical pointing at a host that redirects?

Start here, because it is the most common cause and the hardest to see. In a browser both hosts work: type either one and a page renders. The defect is only visible to something that reads addresses instead of pages.

In Next.js the split has a precise mechanism. metadataBase, and whatever BASE_URL constant you feed it, produce the canonical tag, the hreflang alternates and every <loc> in the sitemap. The host your platform actually serves is decided in a hosting dashboard, and nothing checks that the two agree. Setting www as the primary domain while the code hardcodes the apex is a two-click mistake that survives every build, test and Lighthouse run.

With next-intl it compounds. The middleware builds its Link: rel="alternate" hreflang header from the request's x-forwarded-host, so the header advertises www while the HTML <link> elements advertise the apex. The page then ships two contradictory hreflang clusters. I set alternateLinks: false and left the HTML as the single source of truth, which is the version I can actually inspect with curl.

The fix is a decision, not a refactor. Pick one host, make the other 308 to it, and derive every absolute URL in the codebase from one constant that reads from the environment.

// Must match the host production serves on, including the www/apex choice.
export const BASE_URL = (
  process.env.NEXT_PUBLIC_SITE_URL || 'https://guptaaviral.com'
).replace(/\/+$/, '');

/** Host without protocol — for JSON-LD and for the IndexNow payload. */
export const SITE_HOST = BASE_URL.replace(/^https?:\/\//, '');

Are Accept-Language redirects moving Bingbot off your canonical URLs?

This is the second cause, and on a multilingual Next.js site it is nearly universal. next-intl's localeDetection defaults to true. With it on, the middleware inspects the Accept-Language header and 307-redirects to a locale-prefixed path. A request that says it prefers German never reaches /; it lands on /de. And it is not only the homepage — /services, /about, every unprefixed path behaves the same way.

Two things make it worse than it sounds. The redirect is a 307, so it is temporary and no ranking signal consolidates anywhere. And it usually carries no Vary: Accept-Language, so a shared cache is free to serve one language's redirect to a request that asked for another. A crawler that gets a 307 to /de on your canonical URL reports exactly what mine did: the canonical is a redirect.

The fix is one line, and the pattern it replaces is better anyway. Turn server-side detection off so every URL answers as addressed, let hreflang do the language routing for search engines, and do locale suggestion on the client where a human can dismiss it.

export const routing = defineRouting({
  locales: ['en', 'ar', 'de', 'fr', 'es', 'hi'],
  defaultLocale: 'en',
  localePrefix: 'as-needed',
  alternateLinks: false,   // HTML <link> is the single hreflang source
  localeDetection: false   // no 307 off the canonical URL, ever
});

The related bug: Set-Cookie on a cacheable HTML response

While you are in the middleware, check whether it writes a cookie onto page responses. A geo cookie set on HTML is either baked into the shared CDN cache entry and replayed to everyone, crawlers included, or it stops the response being cached at all — so every crawl becomes an origin invocation. Move the suggestion to an uncached route handler under /api/.

Why does a Bingbot group in robots.txt cancel your wildcard rules?

Because that is what the standard says. Under RFC 9309, a crawler obeys exactly one group — the most specific one whose user-agent matches — and never the union of several. Most people write robots.txt as though the groups accumulate. They do not.

So this file, which looks protective, tells Bing that the admin area is fair game and tells it nothing else:

User-agent: *
Disallow: /admin/

User-agent: Bingbot
Allow: /

Sitemap: https://www.example.com/sitemap.xml

The inverse is the version that hurts an indexing effort: any Disallow you add to the wildcard group six months from now silently will not apply to Bing, and any Allow carve-out you add for Bing will not apply to Google. Two files drifting apart inside one file.

• If you have no crawler-specific rules, delete the named groups. The default state is "allowed" — listing a bot to permit it buys nothing.
• If you want named groups anyway (I keep them, because this site sells crawler readiness and the list is a demonstration), generate every group from one rule object in robots.ts so they cannot drift.
• Sitemap: is a non-group directive. It applies globally no matter which group a crawler picks.
• Bing honours Crawl-delay where Google ignores it. Do not add one to make a file look tidy — it only throttles Bing.
• Bing does support X-Robots-Tag, including noindex and nofollow. Check your headers as well as your meta tags.

What else keeps a technically healthy site out of Bing?

You never verified the property

No verification means no sitemap submission, no URL inspection, no crawl diagnostics and no quota. The fastest route is a one-click import from Google Search Console at Bing Webmaster Tools. The most durable is a DNS CNAME record, which covers the whole domain including subdomains. The file method, BingSiteAuth.xml in public/, works too.

Your content only exists after JavaScript runs

This is the failure mode everyone expects and, in a well-built App Router site, the one least likely to be at fault — server components render text into the HTML by default. It is still worth ten seconds of checking, because Bing is materially less tolerant here than Google, and one 'use client' boundary in the wrong place can hollow out a page.

Check it with view-source or curl, never with browser DevTools — the Elements panel shows you the DOM after hydration, which is the one thing the crawler does not have.

A CDN or WAF is quietly challenging the crawler

A rule that issues a JavaScript challenge or a 403 to anything that looks automated will remove you from Bing without a single error in your application logs. Verified Bingbot sits on most platforms' good-bot allowlists by default, but a custom rule or an aggressive "block non-browser traffic" toggle overrides that. Allowlist by reverse DNS lookup, never by user-agent string.

The domain is new, unlinked and thin

Bing is pickier than Google about fresh domains. "Discovered but not crawled" is the default state for a site nobody links to, and a handful of routes with no external references is the profile Bing declines to spend crawl budget on. No configuration change fixes this one.

How do I map the symptom in Bing Webmaster Tools to the cause?

Bing reports symptoms, not causes, and one message can come from three different bugs. This is the table I work from.

How do you wire IndexNow into a Next.js deploy?

IndexNow turns discovery from a waiting game into a push. One HTTP POST tells Bing, Yandex, Seznam, Naver and Yep that a set of URLs changed, and any participating endpoint shares the submission with the others. DuckDuckGo inherits it downstream of Bing. Google has no equivalent open endpoint — there you still submit in Search Console. The protocol is documented at indexnow.org and fits in one file.

1. 1

   Generate a key

   Between 8 and 128 characters from a-zA-Z0-9-. Thirty-two hex characters is the convention. It is public by design — ownership is proven by serving it, not by hiding it.

2. 2

   Serve the key file

   Put public/<key>.txt in the repo containing exactly the key and nothing else. The match is byte-exact, so no trailing prose and no HTML wrapper. Static hosting from public/ is the most reliable form.

3. 3

   Build the payload

   Four fields: host, key, keyLocation and urlList. Every URL in urlList must be on host — one stray preview URL rejects the entire batch.

4. 4

   POST it

   To https://api.indexnow.org/IndexNow with Content-Type: application/json; charset=utf-8. Read the status code, not the body; the body is usually empty.

5. 5

   Call it from the deploy

   A script nobody remembers to run is not a pipeline. Wire it into a post-deploy step, guarded on the production environment so preview builds never submit.

import {BASE_URL, SITE_HOST} from './constants';

const ENDPOINT = 'https://api.indexnow.org/IndexNow';
const MAX_BATCH = 10_000; // protocol cap for a single request

/** Public by design: ownership is proven by serving the same value at /<key>.txt */
const KEY = process.env.INDEXNOW_KEY ?? '';

export async function submitToIndexNow(urls: string[]) {
  // A mixed-host batch is rejected whole with 422, so drop strays first.
  const onHost = urls.filter((url) => {
    try {
      return new URL(url).host === SITE_HOST;
    } catch {
      return false;
    }
  });
  if (!KEY || onHost.length === 0) return {ok: false, status: 0, submitted: 0};

  const batch = onHost.slice(0, MAX_BATCH);
  const response = await fetch(ENDPOINT, {
    method: 'POST',
    headers: {'Content-Type': 'application/json; charset=utf-8'},
    body: JSON.stringify({
      host: SITE_HOST,
      key: KEY,
      keyLocation: `${BASE_URL}/${KEY}.txt`,
      urlList: batch
    })
  });

  // 200 accepted; 202 accepted with the key still being validated.
  const ok = response.status === 200 || response.status === 202;
  return {ok, status: response.status, submitted: ok ? batch.length : 0};
}

How do you prove each fix actually landed?

Every claim above is checkable from a terminal in under a minute. Run these against production, not a preview URL, and run them before you touch Bing Webmaster Tools — most of what looks like a Bing problem is visible in an HTTP header first.

# 1. Does the host in your canonical and sitemap answer 200, or redirect?
curl -sI -A "bingbot/2.0" https://guptaaviral.com/ | head -2

# 2. Does a non-English request get bounced off the canonical URL?
curl -sI -H "Accept-Language: de-DE,de;q=0.9" https://guptaaviral.com/services | head -2

# 3. Does the canonical in the HTML match the host you just fetched?
curl -s https://guptaaviral.com/ | grep -o '<link rel="canonical"[^>]*>'

# 4. Is every URL in the sitemap a 200? This prints only the ones that are not.
curl -s https://guptaaviral.com/sitemap.xml | grep -o '<loc>[^<]*' | cut -c6- | while read -r u; do echo "$(curl -sI -o /dev/null -w '%{http_code}' "$u") $u"; done | grep -v '^200'

# Key file: 200, text/plain, body identical to the filename.
curl -sI https://guptaaviral.com/40dfb2c08506099e33d49a802495d813.txt | head -3
curl -s  https://guptaaviral.com/40dfb2c08506099e33d49a802495d813.txt

# No Set-Cookie on cacheable HTML, and no unexpected Location.
curl -sI -A "bingbot/2.0" https://guptaaviral.com/services | grep -iE 'HTTP/|location|set-cookie|vary'

# Which group would Bingbot obey? Read it, do not assume it.
curl -s https://guptaaviral.com/robots.txt

Then move to Bing Webmaster Tools and confirm four things in order: the verified property is the exact host that answers 200; the sitemap reports every URL discovered with zero redirect errors; URL Inspection says the homepage is indexable and the live fetch shows real text; and the IndexNow tab shows the counts you submitted. If those agree, the technical work is done and the rest is authority and content.

Why is the site still not indexed after all of this?

Because submission is not indexing, and Bing has never pretended otherwise. IndexNow queues a URL for crawling. It does not promise a crawl, and a crawl does not promise an index entry. On a domain with no history, a submitted URL can sit pending for days while Bing decides whether the site is worth the bandwidth.

At that point you are out of engineering problems and into two ordinary ones. The first is links: two or three genuine external references move a domain off "Crawl Pending" faster than any amount of resubmission. The second is substance — a site with a dozen thin routes gives Bing nothing to rank, and "Discovered but not crawled" fires on low-value pages as readily as on unreachable ones.

Freshness is the other lever worth knowing about, because it applies to both search and AI answers: 2026 citation research from Digital Applied found roughly half of AI citations go to content published in the last thirteen weeks. Publishing regularly and pinging IndexNow on publish is a far better use of the protocol than resubmitting an unchanged sitemap.

None of this is exotic work. It is four configuration decisions, one small library file and a deploy hook — and it is the same class of problem I fix during a Next.js build or a migration, where the difference between a canonical that resolves and one that redirects is the difference between keeping your rankings and starting over. If you would like me to run these checks against your domain, get in touch and I will do an introductory project call.