Seeking help! No matter how many keywords/names u search related to this story the article doesn't appear on Google
This was a front page A1 story I wrote for WaPo on how smear campaigns and abuse women journalists endure are a press freedom issue. Can someone explain why the article does not appear on Google? https://www.washingtonpost.com/investigations/2023/02/14/women-journalists-global-violence/
@taylorlorenz not sure this is what may be playing a role, but WaPo’s robots.txt file (one way webmasters can request robots handle things on their domain) has this:
User-agent: Google-Extended
Disallow: /
That is requesting any machine that makes a request with the user-agent set to “Google-Extended” not access (and therefore not index) anything at all on that domain.
Odd thing to have in a robots.txt file, but it is there at https://www.washingtonpost.com/robots.txt
@davidaugust @taylorlorenz Google extended is the ai search crawler, normal search still uses Google it which is allowed by the robots file
Google however has no index of the URL, which could be done by either Google in the background, or by wapo themselves in the search console, and with Google making it very difficult to submit URLs that you don't own, you would probably have to ask wapo to submit it for crawling manually to try and fix it
@tristan @taylorlorenz I agree. The WaPo web team (or even search marketing or organic search team, not sure how WaPo is structured) may be able to get into it as Tristan says.