The noindex that robots.txt was hiding

This site serves a resume PDF. It carries personal contact details, so it must never be a search landing page — and the obvious move is one line in robots.txt:

Disallow: /resume/

That is the wrong fix, and it is wrong in a way that produces the exact outcome it was meant to prevent.

#Link to this sectionDisallow is not a removal instruction

Disallow tells a crawler not to fetch a URL. It says nothing about whether the URL may appear in results. If Google learns a URL some other way — and it will, because the navbar and the mobile menu on this site link that PDF from every page — it can index the address with no content behind it. That is the "Indexed, though blocked by robots.txt" state, and it is worse than being indexed normally: the result exists, and you have no way to influence what it says.

The trap is that the two mechanisms are ordered. noindex is a directive on the resource. To obey it, the crawler has to fetch the resource. Block the fetch and the directive is never read. You have not suppressed the page; you have suppressed the instruction that would have suppressed it.

#Link to this sectionWhat actually works on a PDF

A PDF has nowhere to put <meta name="robots">, so the directive has to travel in a header. In next.config.ts:

async headers() {
  return [
    {
      source: "/resume/:path*",
      headers: [{ key: "X-Robots-Tag", value: "noindex" }],
    },
  ];
}

Then leave the path crawlable. The crawler follows the link, fetches the file, reads noindex in the response headers, and drops it. The block is removed so that the instruction can be delivered.

#Link to this sectionThe general shape

Every time you want something gone from an index, ask which of the two you are doing: preventing a fetch, or issuing an instruction. They compose badly, because the second one is delivered by the first. If a URL is linked from anywhere you control, a Disallow is a way of guaranteeing you never get to say what you wanted to say about it.