Dynamic URL and headers

Using this feature of dynamic URL and headers, you can dynamically create a source URL and scrap content on the fly. Basically this lets you fetch content from a single underlying source by passing multiple get arguments to it. This feature will convert specific text in your URL or headers arguments to corresponding value based on some placeholders.

For example, if you want a page to scrap symbols on reuters.com dynamically based on user input then:

  • url should be http://www.reuters.com/finance/stocks/overview?symbol=___symbol___
  • get argument for page should be `http://yourdomain.com/page/?symbol=CSCO.O (to get Cisco details)

This will replace ___symbol___ in the url with CSCO.O in realtime. You can use multiple such replacement variables in your url or postargs. Such replacement variables should be wrapped between 3 underscores. Note that field names being passed this was are case-sensitive. Having ‘FieldName’ vs. ‘fieldname’ makes a difference.

You can also use the special variable ___QUERY_STRING___ to replace the complete query string post.

Check this example code, where the get argument ‘symbol’ is used to dynamically build a URL and fetch content.

Leave a Reply