One to url them all…

WordPress announced sitemaps support, which I though might be a chance for me to get all the post – not just last 10… so wouldn’t have to do cyclic rss parsing…

but no (here’s  mine), it’s just the permalink’s list… no author, tag/category, summary info that goes along with atom…

So then I thought the pretty permalinks + atom is the answer, i.e.: these are my first ten posts here:

https://marekopel.wordpress.com/2006/11/page/2

and atom for another 10 posts (last 10 from November):

https://marekopel.wordpress.com/2006/11/feed/atom

so the answer to my problem (i.e. atom for first 10 posts) would be:

https://marekopel.wordpress.com/2006/11/page/2/feed/atom

…but it won’t work… why? ;(

still I can go through each day of the calendar, i.e.:

https://marekopel.wordpress.com/2006/11/5/feed/atom

– hopefully you won’t get more than 10 posts a day…

but now I can see I can’t compete with GGL and index all the wordpress.com

once I was moaning about getting only a few thousands of crawled blogs using Next link, when there are hundreds of thousands of WP blogs created each month

now I think I gained the critical mass, and parsing only blogrolls (also the non-XFN ones) I got:

  • 2006.06.15: 25 996 WP blog URLs
  • a few days later: 55 689 WP blog URLs

and I only managed to parse half of it…

I can have 1000 WP blog’s atoms parsed in ~6 days… even when going parallel  (say 5 sessions – my server can handle that🙂 -> when I finish parsing the last thousand the first one is already outdated …

I hoped to experiment comparing my search engine (based on Solr – more details soon) to GGL Blog Search in a given period of time (say a week)… now even that seems impossible… what to do? what to do? <panic>

Tagi: ,

Skomentuj

Please log in using one of these methods to post your comment:

Logo WordPress.com

Komentujesz korzystając z konta WordPress.com. Log Out / Zmień )

Zdjęcie z Twittera

Komentujesz korzystając z konta Twitter. Log Out / Zmień )

Facebook photo

Komentujesz korzystając z konta Facebook. Log Out / Zmień )

Google+ photo

Komentujesz korzystając z konta Google+. Log Out / Zmień )

Connecting to %s


%d bloggers like this: