{"id":1661,"date":"2026-09-23T11:13:38","date_gmt":"2026-09-23T10:13:38","guid":{"rendered":"https:\/\/ruby-doc.org\/learn\/?p=1661"},"modified":"2026-09-24T12:00:36","modified_gmt":"2026-09-24T11:00:36","slug":"build-a-reliable-ruby-http-fetcher-with-nethttp","status":"publish","type":"post","link":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/","title":{"rendered":"Build a Reliable Ruby HTTP Fetcher with Net::HTTP"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">A scraper that worked on Friday and fails on Monday may have nothing wrong with its parser. The page might now be a login form, the proxy credentials could have expired, or the server could be asking the client to slow down. Those failures become harder to diagnose when every caller handles the network request differently.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For a Ruby batch job, a small shared fetch method is often enough to bring that behaviour into one place. This example uses <code>Net::HTTP<\/code> and <code>URI<\/code>, accepts an optional HTTP proxy, and returns the information the job needs before it starts parsing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Give the job a useful response<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The method returns a <code>FetchResult<\/code> containing the status code, headers and body. Receiving a 403 or 429 still counts as receiving a response; the caller gets to decide what happens next. A connection failure takes the exception path instead.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">That leaves a fairly simple division of work. The fetcher deals with the HTTP request. The job knows whether a particular response is useful, whether a retry is justified and how long the whole import is allowed to run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Save this as <code>fetch.rb<\/code>. It uses APIs documented for Ruby 3.4, with the required standard-library gems installed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>require \"net\/http\"\nrequire \"uri\"\nrequire \"openssl\"\nrequire \"zlib\"\nrequire \"logger\"\nrequire \"json\"\n\nFetchResult = Struct.new(:code, :headers, :body, keyword_init: true)\nclass FetchError &lt; StandardError; end\n\ndef fetch(url, proxy: nil, headers: {}, timeout: 15, logger: nil)\n  target = URI.parse(url)\n  unless target.is_a?(URI::HTTP) &amp;&amp; target.hostname &amp;&amp;\n         !target.userinfo &amp;&amp; (1..65_535).cover?(target.port)\n    raise ArgumentError, \"expected an HTTP(S) URL without credentials\"\n  end\n\n  seconds = Float(timeout)\n  unless seconds.finite? &amp;&amp; seconds.positive?\n    raise ArgumentError, \"timeout must be positive and finite\"\n  end\n\n  gateway = proxy ? URI.parse(proxy) : nil\n  if gateway\n    unless gateway.scheme == \"http\" &amp;&amp; gateway.hostname &amp;&amp;\n           (1..65_535).cover?(gateway.port) &amp;&amp;\n           &#91;\"\", \"\/\"].include?(gateway.path) &amp;&amp;\n           !gateway.query &amp;&amp; !gateway.fragment\n      raise ArgumentError, \"expected an HTTP proxy URL\"\n    end\n\n    decode = -&gt;(value) { value &amp;&amp; URI::DEFAULT_PARSER.unescape(value) }\n    http = Net::HTTP.new(\n      target.hostname, target.port, gateway.hostname, gateway.port,\n      decode.call(gateway.user), decode.call(gateway.password)\n    )\n  else\n    http = Net::HTTP.new(target.hostname, target.port, nil)\n  end\n\n  http.open_timeout = seconds\n  http.read_timeout = seconds\n  http.write_timeout = seconds\n  http.max_retries = 0\n  http.use_ssl = target.scheme == \"https\"\n  if http.use_ssl?\n    http.min_version = OpenSSL::SSL::TLS1_2_VERSION\n    http.verify_mode = OpenSSL::SSL::VERIFY_PEER\n    http.verify_hostname = true\n  end\n\n  request = Net::HTTP::Get.new(target, headers)\n\n  context = { host: target.hostname, proxy_host: gateway&amp;.hostname }\n  begin\n    response = http.request(request)\n  rescue Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout,\n         SocketError, SystemCallError, IOError,\n         OpenSSL::SSL::SSLError, Net::ProtocolError,\n         Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Zlib::Error =&gt; error\n    logger&amp;.warn(JSON.generate(context.merge(\n      event: \"fetch_error\", code: nil, bytes: 0, error: error.class.name\n    )))\n    raise FetchError, \"HTTP fetch failed\", cause: error\n  end\n\n  body = response.body || \"\"\n  logger&amp;.info(JSON.generate(context.merge(\n    event: \"fetch\", code: response.code.to_i, bytes: body.bytesize\n  )))\n  FetchResult.new(code: response.code.to_i, headers: response.to_hash, body: body)\nend<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">There is no custom implementation of <code>cause<\/code> here. Ruby already supports exception chaining through <code>raise ... cause: error<\/code>. Configuration errors are also allowed to surface before the request, rather than being caught by a broad <code>rescue StandardError<\/code> and reported as network trouble.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The rescue clause covers common transport, protocol and decompression failures. Other exceptions can still escape it.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Why the proxy argument matters<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The direct connection branch contains an easy detail to miss:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>Net::HTTP.new(target.hostname, target.port, nil)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">That final <code>nil<\/code> disables proxy discovery from the environment. Leaving it out allows <code>Net::HTTP<\/code> to pick up <code>http_proxy<\/code>. This can explain why the same script takes different routes on a laptop and a scheduled worker. Here, a caller passing <code>proxy: nil<\/code> gets a direct connection.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The other branch accepts an HTTP proxy. <a href=\"https:\/\/www.ibm.com\/docs\/en\/b2b-integrator\/6.2.2?topic=destinations-configuring-http-https-destination\">For an HTTPS destination<\/a>, the connection uses a CONNECT tunnel, with TLS protecting traffic to the target. SOCKS and TLS to the proxy itself are outside this example.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/ruby-doc.org\/3.3.6\/stdlibs\/net\/Net\/HTTP.html\">Ruby supplies port 80<\/a> when an HTTP proxy URL <a href=\"https:\/\/webmasters.stackexchange.com\/questions\/127105\/is-http-in-url-referring-to-80-port-number\">omits its port<\/a>. A provider using 8080 or another port needs that value included in the URL. Consequently, checking <code>URI#port<\/code> alone cannot tell you whether the original string specified a port.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Credentials belong in deployment configuration or a secret store. The helper decodes percent-encoded usernames and passwords before passing them to <code>Net::HTTP<\/code>; the full proxy URL is never included in its log message.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Check what came back<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A caller can use the helper like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>require_relative \"fetch\"\n\nresult = fetch(\n  \"https:\/\/example.com\/\",\n  proxy: ENV&#91;\"FETCH_PROXY_URL\"],\n  headers: { \"Accept\" =&gt; \"text\/html\" },\n  logger: Logger.new($stdout)\n)\n\nputs result.code\nputs result.headers&#91;\"content-type\"]&amp;.first\nputs result.body.bytesize<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">For a normal 200 HTML response, those lines print <code>200<\/code>, its content type and the body&#8217;s size in bytes. The last two values depend on the server. The header hash uses lower-case names and arrays of values.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An HTML parser will quite happily parse an access-denied page. Checking the status and expected content before parsing helps prevent an empty import from looking like a successful run.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The helper returns redirects to the caller, too. It has no automatic redirect following, cookie jar or connection reuse between calls.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What the timeout actually limits<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The <code>timeout<\/code> argument is applied to opening, reading and writing. It does not put a 15-second ceiling on the entire request: a read timeout limits an individual blocking read. A server that keeps sending small chunks may keep the transfer running much longer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The import therefore needs its own time budget. A short retry after an opening or reading timeout may be reasonable for a GET, but repeated attempts still consume that budget. Certificate failures need investigation; turning off verification would hide the problem.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Setting <code>max_retries = 0<\/code> disables the client&#8217;s built-in retry mechanism so that the job can own this decision. HTTPS certificate and hostname verification remain enabled, with TLS 1.2 as the minimum protocol version.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Different responses call for different handling:<\/p>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Result<\/th><th>What the job should consider<\/th><\/tr><\/thead><tbody><tr><td>Expected 200 response<\/td><td>Check the content, then pass it to the parser.<\/td><\/tr><tr><td>403 Forbidden<\/td><td>Investigate permissions, credentials and access requirements.<\/td><\/tr><tr><td>429 Too Many Requests<\/td><td>Respect a usable <code>Retry-After<\/code> value and reduce the request rate.<\/td><\/tr><tr><td>Opening or reading timeout<\/td><td>Retry only within a bounded attempt and time budget.<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">A 429 response may supply <code>Retry-After<\/code>. That value can be seconds or an HTTP date. When its delay is longer than the worker can wait, rescheduling preserves the server&#8217;s requested pause. Without a usable value, bounded backoff with jitter helps keep multiple workers from retrying together.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A 403 response means the server understood the request but refuses to fulfil it. It is not, by itself, evidence that the proxy needs replacing.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Leave ordinary gzip handling to Net::HTTP<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The example leaves <code>Accept-Encoding<\/code> alone. With zlib support available, <code>Net::HTTP<\/code> normally negotiates supported compression and decompresses the response. Adding a second gzip decoder can mean trying to decompress content that has already been decoded.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A caller that sets <code>Accept-Encoding<\/code> itself takes responsibility for the changed decoding behaviour. Range requests also need separate treatment.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This matters when reading the logs: <code>body.bytesize<\/code> measures the returned body. After automatic decompression, it tells you how many decoded bytes the parser receives, rather than how many bytes crossed the network.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Keep related requests on the same session<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Some authorised workflows associate a login with an IP address. Choosing a random proxy for every request can disrupt them. The job can instead select a proxy once and pass it through the related calls, while a separate cookie jar holds that session&#8217;s cookies.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A stable proxy URL does not always guarantee a stable exit IP. A provider&#8217;s rotating endpoint may need its own session identifier. Equally, keeping the same IP does not create a cookie jar in Ruby.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Where a wider workflow includes a cloud phone, Byteful&#8217;s <a href=\"https:\/\/byteful.com\/blog\/geelark-proxy-integration\">GeeLark: Proxy Setup for the Antidetect Cloud Phone<\/a> covers that product&#8217;s configuration. This fetcher remains a separate Ruby HTTP client; it does not reproduce a mobile browser or device session.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If the job grows into a pool of sessions, expiry and concurrency become part of the design. A monotonic clock is suitable for measuring elapsed time, and expired mappings need removing so old session keys do not accumulate indefinitely.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Make the logs useful without storing the response<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The helper records the target host, proxy host, status and returned byte count. Its failure log contains an exception class. That is enough to distinguish several common problems without placing passwords, cookies, bodies or full URLs in routine logs.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Query strings can contain tokens or personal information. Even exception messages may include connection details, which is why the example logs the class rather than the message. The underlying exception remains accessible through <code>FetchError#cause<\/code> for controlled diagnosis.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">There are two limits to address before expanding this into a general-purpose service. First, the whole body is read into memory; large or unpredictable responses need streaming with a maximum decoded size. Second, accepting arbitrary user-supplied URLs requires destination restrictions and DNS\/IP checks. Scheme validation alone does not prevent server-side request forgery.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For a scheduled collection job, approved destinations, request rates and retention rules should live alongside its configuration. The target&#8217;s published access requirements and robots guidance belong in that review. From there, an observed problem can guide the next change, whether that is connection reuse, a shared rate limiter or a response-size limit.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A scraper that worked on Friday and fails on Monday may have nothing wrong with its parser. The page might now be a login form, the proxy credentials could have expired, or the server could be asking the client to slow down. Those failures become harder to diagnose when every caller handles the network request [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":1662,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[],"class_list":["post-1661","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ruby-tips"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":7}},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.5 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Build a Reliable Ruby HTTP Fetcher with Net::HTTP - Ruby-Doc Learn<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Build a Reliable Ruby HTTP Fetcher with Net::HTTP - Ruby-Doc Learn\" \/>\n<meta property=\"og:description\" content=\"A scraper that worked on Friday and fails on Monday may have nothing wrong with its parser. The page might now be a login form, the proxy credentials could have expired, or the server could be asking the client to slow down. Those failures become harder to diagnose when every caller handles the network request [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/\" \/>\n<meta property=\"og:site_name\" content=\"Ruby-Doc Learn\" \/>\n<meta property=\"article:published_time\" content=\"2026-09-23T10:13:38+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-09-24T11:00:36+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/reliable-http-requests-in-ruby.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1672\" \/>\n\t<meta property=\"og:image:height\" content=\"941\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"James Britt\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"James Britt\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/\"},\"author\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/james-britt\\\/#james-britt\"},\"headline\":\"Build a Reliable Ruby HTTP Fetcher with Net::HTTP\",\"datePublished\":\"2026-09-23T10:13:38+00:00\",\"dateModified\":\"2026-09-24T11:00:36+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/\"},\"wordCount\":1195,\"publisher\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/reliable-http-requests-in-ruby.png\",\"articleSection\":[\"Ruby tips\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/\",\"name\":\"Build a Reliable Ruby HTTP Fetcher with Net::HTTP - Ruby-Doc Learn\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/reliable-http-requests-in-ruby.png\",\"datePublished\":\"2026-09-23T10:13:38+00:00\",\"dateModified\":\"2026-09-24T11:00:36+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/reliable-http-requests-in-ruby.png\",\"contentUrl\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/reliable-http-requests-in-ruby.png\",\"width\":1672,\"height\":941,\"caption\":\"reliable http requests in ruby\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/build-a-reliable-ruby-http-fetcher-with-nethttp\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Build a Reliable Ruby HTTP Fetcher with Net::HTTP\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#website\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/\",\"name\":\"Ruby-Doc Learn\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#organization\",\"name\":\"Ruby-Doc Learn\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/ruby-doc-logo-transparent.png\",\"contentUrl\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/ruby-doc-logo-transparent.png\",\"width\":1650,\"height\":305,\"caption\":\"Ruby-Doc Learn\"},\"image\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/james-britt\\\/#james-britt\",\"name\":\"James Britt\",\"image\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/wp-content\\\/uploads\\\/2026\\\/09\\\/jgb_self-portrait-20140914.png\",\"description\":\"Creator and long-term maintainer of Ruby-Doc.org, founder of Neurogami, and author of practical Ruby tutorials for Ruby-Doc Learn.\",\"sameAs\":[\"https:\\\/\\\/jamesbritt.com\\\/\",\"https:\\\/\\\/neurogami.com\\\/\",\"https:\\\/\\\/www.oreilly.com\\\/pub\\\/au\\\/2595\",\"https:\\\/\\\/www.rubyevents.org\\\/profiles\\\/james-britt\"],\"url\":\"https:\\\/\\\/ruby-doc.org\\\/learn\\\/james-britt\\\/\",\"jobTitle\":\"Ruby developer and writer\",\"knowsAbout\":[\"Ruby programming language\",\"Ruby documentation\",\"JRuby\",\"Open Sound Control\",\"Software development\"],\"affiliation\":{\"@type\":\"Organization\",\"name\":\"Ruby-Doc.org\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Build a Reliable Ruby HTTP Fetcher with Net::HTTP - Ruby-Doc Learn","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/","og_locale":"en_US","og_type":"article","og_title":"Build a Reliable Ruby HTTP Fetcher with Net::HTTP - Ruby-Doc Learn","og_description":"A scraper that worked on Friday and fails on Monday may have nothing wrong with its parser. The page might now be a login form, the proxy credentials could have expired, or the server could be asking the client to slow down. Those failures become harder to diagnose when every caller handles the network request [&hellip;]","og_url":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/","og_site_name":"Ruby-Doc Learn","article_published_time":"2026-09-23T10:13:38+00:00","article_modified_time":"2026-09-24T11:00:36+00:00","og_image":[{"width":1672,"height":941,"url":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/reliable-http-requests-in-ruby.png","type":"image\/png"}],"author":"James Britt","twitter_card":"summary_large_image","twitter_misc":{"Written by":"James Britt","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#article","isPartOf":{"@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/"},"author":{"@id":"https:\/\/ruby-doc.org\/learn\/james-britt\/#james-britt"},"headline":"Build a Reliable Ruby HTTP Fetcher with Net::HTTP","datePublished":"2026-09-23T10:13:38+00:00","dateModified":"2026-09-24T11:00:36+00:00","mainEntityOfPage":{"@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/"},"wordCount":1195,"publisher":{"@id":"https:\/\/ruby-doc.org\/learn\/#organization"},"image":{"@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#primaryimage"},"thumbnailUrl":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/reliable-http-requests-in-ruby.png","articleSection":["Ruby tips"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/","url":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/","name":"Build a Reliable Ruby HTTP Fetcher with Net::HTTP - Ruby-Doc Learn","isPartOf":{"@id":"https:\/\/ruby-doc.org\/learn\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#primaryimage"},"image":{"@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#primaryimage"},"thumbnailUrl":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/reliable-http-requests-in-ruby.png","datePublished":"2026-09-23T10:13:38+00:00","dateModified":"2026-09-24T11:00:36+00:00","breadcrumb":{"@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#primaryimage","url":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/reliable-http-requests-in-ruby.png","contentUrl":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/reliable-http-requests-in-ruby.png","width":1672,"height":941,"caption":"reliable http requests in ruby"},{"@type":"BreadcrumbList","@id":"https:\/\/ruby-doc.org\/learn\/build-a-reliable-ruby-http-fetcher-with-nethttp\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ruby-doc.org\/learn\/"},{"@type":"ListItem","position":2,"name":"Build a Reliable Ruby HTTP Fetcher with Net::HTTP"}]},{"@type":"WebSite","@id":"https:\/\/ruby-doc.org\/learn\/#website","url":"https:\/\/ruby-doc.org\/learn\/","name":"Ruby-Doc Learn","description":"","publisher":{"@id":"https:\/\/ruby-doc.org\/learn\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ruby-doc.org\/learn\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ruby-doc.org\/learn\/#organization","name":"Ruby-Doc Learn","url":"https:\/\/ruby-doc.org\/learn\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ruby-doc.org\/learn\/#\/schema\/logo\/image\/","url":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/ruby-doc-logo-transparent.png","contentUrl":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/ruby-doc-logo-transparent.png","width":1650,"height":305,"caption":"Ruby-Doc Learn"},"image":{"@id":"https:\/\/ruby-doc.org\/learn\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ruby-doc.org\/learn\/james-britt\/#james-britt","name":"James Britt","image":"https:\/\/ruby-doc.org\/learn\/wp-content\/uploads\/2026\/09\/jgb_self-portrait-20140914.png","description":"Creator and long-term maintainer of Ruby-Doc.org, founder of Neurogami, and author of practical Ruby tutorials for Ruby-Doc Learn.","sameAs":["https:\/\/jamesbritt.com\/","https:\/\/neurogami.com\/","https:\/\/www.oreilly.com\/pub\/au\/2595","https:\/\/www.rubyevents.org\/profiles\/james-britt"],"url":"https:\/\/ruby-doc.org\/learn\/james-britt\/","jobTitle":"Ruby developer and writer","knowsAbout":["Ruby programming language","Ruby documentation","JRuby","Open Sound Control","Software development"],"affiliation":{"@type":"Organization","name":"Ruby-Doc.org","url":"https:\/\/ruby-doc.org\/"}}]}},"_links":{"self":[{"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/posts\/1661","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/comments?post=1661"}],"version-history":[{"count":2,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/posts\/1661\/revisions"}],"predecessor-version":[{"id":1716,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/posts\/1661\/revisions\/1716"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/media\/1662"}],"wp:attachment":[{"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/media?parent=1661"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/categories?post=1661"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ruby-doc.org\/learn\/wp-json\/wp\/v2\/tags?post=1661"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}