“IndexNow” is an automatic way to get participating search engines to come search specified URLs your site, and one cool thing about it is if you submit an URL to one, it gets pushed to all. I wanted an easy and automatic way to submit if there are any new URLs on this blog, so I started thinking of a way it could be done. Let me share that with you. 🤓

I have:

  • A json feed https://cogley.jp/feed.json with post dates and URLs
  • A webhook.site subscription that lets me schedule custom scripts that can handle json

… so I should be able to parse my json, and ping indexnow for any URLs added in the last day, so that if I schedule a daily script run, it will submit just those.

To get set up I followed the steps over on Bing’s indexnow page. First you need to create a file on your site. In micro.blog I edited my custom theme, and added a new template under static. If the API key is 47e6f02ccea743bf , then you create a file called 47e6f02ccea743bf.txt, and add 47e6f02ccea743bf to the body of the file.

The existence of this file serves as proof you own the site, and once published, should be available at:

https://mysite.com/47e6f02ccea743bf.txt

Now when you have a new or updated URL, you can ping by accessing this API:

https://www.bing.com/indexnow?url=https://mysite.com/path/to/newpost.html&key=47e6f02ccea743bf

As for automation, I tried various things, but in the end got help from Simon, the owner of webhook.site. You can set up a script there, like this:

 
feed = request('https://mysite.com/feed.json')
json = json_decode(feed['content'])

for (item in json['items']) {
    interval = date_interval(item['date_published'], now())
    
    if (interval < 86400) {
        res = request(
            'https://www.bing.com/indexnow?key=47e6f02ccea743bf&url={}'.format(url_encode(item['url']))
        )
        echo('Submitted {}; status: {}'.format(item['url'], res['status']))

    }
} 
request('https://pingomatic.com/ping/?title=My+Blog&blogurl=https%3A%2F%2Fmysite.com&rssurl=http%3A%2F%2Fmysite.com%2Ffeed.xml&chk_blogs=on&chk_feedburner=on&chk_tailrank=on&chk_superfeedr=on')

This assumes:

  • your feed.json is structured like mine
  • you registered the api key file so indexnow can find it
  • you also want to ping “ping-o-matic” about the presence of changes

What the script does is, loops through the “items” node checking all the “date published” values against the difference between now and that date, and if the interval is less than 86400 (seconds in a day), then it will hit the indexnow API with those URLs. As a last step, it pushes the rss feed to ping-o-matic.

Once it’s working, to trigger it you can schedule it, or, just bookmark and periodically access your webhook.site url, which will look something like: https://webhook.site/#!/a9f864e4-3506-4a13-943d-c5696e9e1eb8. I scheduled mine because I’m updating every day, but if I wasn’t, I’d just bookmark the webhook.site URL and hit that any time I made a new post.

Manton got “lastmod” working on micro.blog posts, so I can now update my json feed to include the modified timestamp, and check also for recently modified posts. I’ll do that and report back.

There are plenty of ways to do the same thing, from shell scripts to iOS shortcuts, probably. Perhaps Google will even participate in this system one day.

Updates

  • 20231108 - Ok, after observing the script for a couple days, it is indeed picking up the new posts as expected, so the schedule and script is working as expected. Updated the wording to make it clear that the script is searching for new posts, not modified ones.
  • 20231109 - Mentioned that modified date is now working so that can be added and tested.