Customizing HTTP Request Headers

Problem

When you make an HTTP request, you want to specify custom HTTP headers like " User-Agent" or " Accept-Language".

Solution

Pass in a Hash of header values to Net::HTTP#get or Net::HTTP#post:

require et/http require uri #A simple wrapper method that accepts either strings or URI objects #and performs an HTTP GET. module Net class HTTP def HTTP.get_with_headers(uri, headers=nil) uri = URI.parse(uri) if uri.respond_to? :to_str start(uri.host, uri.port) do | http| return http.get(uri.path, headers) end end end end #Lets get a web page in German. res = Net::HTTP.get_with_headers(http://www.google.com/, { Accept-Language => de}) #Check a bit of the body to make sure its really in German. s = res.body.size res.body[s-200..s-140] # => "ngebote</a> - <a href=/intl/de/about.html>Alles 374ber Google

Discussion

Usually you can retrieve the web pages you want without specifying any custom HTTP headers. As you start performing more complicated interactions with web servers, youll find yourself customizing the headers more.

For instance, if you write a web spider or client, youll want it to send a " User-Agent" header on every request, identifying itself to the web server. Unlike the HTTP client libraries for other programming languages, the net/http library doesn send a "User-Agent" header by default; its your reponsibility to send one.

Net::HTTP.get_with_headers(url, {User-Agent => Ruby Web Browser v1.0})

You can often save bandwidth (at the expense of computer time) by sending an "Accept-Encoding" header, requesting that a web server compress data before sending it to you. Gzip compression is the most common way a server compresses HTTP response data; you can reverse it with Rubys zlib library:

uncompressed = Net::HTTP.get_with_headers(http://www.cnn.com/) uncompressed.body.size # => 65150 gzipped = Net::HTTP.get_with_headers(http://www.cnn.com/, {Accept-Encoding => gzip}) gzipped[Content-Encoding] # => "gzip" gzipped.body.size # => 14600 require zlib require stringio body_io = StringIO.new(gzipped.body) unzipped_body = Zlib::GzipReader.new(body_io).read() unzipped_body.size # => 65150

If you want to build up a HTTP request with multiple values for the same HTTP header, you can construct a Net:: HTTP::Get (or Net::HTTP::Post) object and call the add_field method multiple times. The example in the Solution used the " Accept-Language" header to request a document in a specific language. The following code fetches the same URL, but its "Accept-Language" header indicates that it will accept a document written in any of four different dialects:

uri = URI.parse(http://www.google.com/) request = Net::HTTP::Get.new(uri.path) [en_us, en, en_gb, ja].each do |language| request.add_field(Accept-Language, language) end request[Accept-Language] # => "en_us, en, en_gb, ja" Net::HTTP.start(uri.host, uri.port) do |http| response = http.request(request) # … process the HTTPResponse object here end

See Also

• Recipe 12.10, "Compressing and Archiving Files with Gzip and Tar," for more about the zlib library
• Recipe 14.1, "Grabbing the Contents of a Web Page"
• Recipe 14.20, "A Real-World HTTP Client," covers a lot of edge cases youll need to handle if you want to write a general-purpose client
• REST web services often use the value of the "Accept" header to provide multiple representations of the same resource; Joe Gregorios article "Should you use Content Negotiation in your Web Services?" explains why its a better idea to provide a different URL for each representation (http://bitworking.org/news/WebServicesAndContentNegotiation)
• Recipe 16.1 for more on REST web services