Diggin’ up the Invisible Web

05Jun07

Sometimes, you happen to discover tons of relevant, unique content lying just below the opaque surface of what we know as the “visible Web”. And sometimes, you may find it only takes a little scratching and scraping to dig up thousands of pages, previously inaccessible to search engines, and thus completely unknown to them. Just a matter of converting a few POSTs to GETs (or vice versa), rewriting a few URLs, and setting up a spider-friendly link structure: technically trivial tasks for most of us.

So, next time, you’d better investigate your clients’ web sites and databases thoroughly before you start building content from scratch: behind those ugly dynamic URLs and obtrusive interfaces that only work with JavaScript and cookie support enabled, a true goldmine might be hidden.

7 Responses to “Diggin' up the Invisible Web”


  1. 1 Petro Posted June 6th, 2007 - 12:16 am

    So true!
    And how lucky was that client, having YOU and not someone else as the author of that discover! It’s a matter of time, but reverse engineering on other’s websites happens… well, to me at least ;-)
    Happy new blog, Everfluxx! I wish you all the best!

    (and thank you for citing me as “respected” - it’s really an honor to me)

  2. 2 Gigi Posted June 9th, 2007 - 11:18 am

    Happy new blog!
    Your first post reminds me of some words Goethe once wrote:
    “So sucht man in dem weiten Sand des Meers/vergebens ein Perle, die verborgen/in stillen Schalen eingeschlossen ruht”. It simply explains what you did discover working almost 24/7 on that project: you uselessly search in the deep sands of the sea a pearl, which calmy lies nested in itself. (the translation sounds more or less so)! But you found it! And that matters :-) See ya!

  3. 3 Matt Kazz Posted June 11th, 2007 - 2:09 am

    This blog reminds me my uncle that said “Parla come magni”. Cheeseburger, wurstel… Help me: spaghetti, pizza, I sing ammore!

  4. 4 Stuart Posted June 11th, 2007 - 11:11 am

    Hello Everfluxx…

    I rarely reply to blogs’ articles (as a user). No reason, I’m just not used to. Btw this time I cant help wishing you the best of luck (and a tbPR >=5 next export :-D).

    I’ll be happy reading you again and again, bro’.

  5. 5 Everfluxx Posted June 12th, 2007 - 10:33 pm

    Yo, comrades! Everfluxx lovez y’all. :D

  6. 6 Paul Posted July 1st, 2007 - 8:14 pm

    I have said this on many times, and believe you too. It is much easier to completely re-design a website from 0, than try to change other one’s work. It’s just a mess if you do it.

    On the other hand, rewriting the URLs or setting up a spider-friendly link structure might well not be enough.

  7. 7 Everfluxx Posted July 1st, 2007 - 8:30 pm

    I agree, Paul. However, sometimes a complete site redesign (which would yield the best results by far) simply is not an option. :/

Leave a Reply


Comment guidelines: No spamming, no profanity, and no flaming. Inappropriate comments will be deleted outright.