Maybe they have a way to unblock major search engine crawlers but block it for everyone else now? I know Cloudflare was doing something similar for some bot protection mechanism, and this seems like something news outlets would want to do also.