{"id":1158,"date":"2026-08-25T08:47:00","date_gmt":"2026-08-25T08:47:00","guid":{"rendered":"https:\/\/ruby-doc.org\/blog\/?p=1158"},"modified":"2026-08-25T08:51:04","modified_gmt":"2026-08-25T08:51:04","slug":"proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs","status":"publish","type":"post","link":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/","title":{"rendered":"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Scrape jobs fail more from ops drift than from parse bugs. Proxies change, targets add rate caps, and TLS rules shift. Ruby ships solid building blocks for a tight fetch layer.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This note defines a small API that fits Ruby core and standard lib style. It keeps method shapes clear, favors explicit args, and works well in batch jobs. Use it as a base for Byteful data pulls where you need repeat runs and clean logs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Define a narrow fetch contract<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Start with one call that returns three fields: status, headers, body. Avoid hidden globals. Route all net errors to one exception type.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Signature<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">FetchResult = Struct.new(:code, :headers, :body, keyword_init: true)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">class FetchError &amp;lt; StandardError<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">attr_reader :cause<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">def initialize(msg, cause: nil)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">@cause = cause<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">super(msg)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"># @param url [String]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"># @param proxy [String, nil] &#8220;http:\/\/user:pass@host:port&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"># @param headers [Hash{String =&amp;gt; String}]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"># @param timeout [Numeric]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"># @return [FetchResult]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">def fetch(url, proxy: nil, headers: {}, timeout: 15)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ruby-doc readers expect small objects and clear args. Use URI for parse and Net::HTTP for I\/O. Keep default timeouts short so jobs fail fast.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Wire Net::HTTP with explicit proxy and TLS<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">TCP ports <a href=\"https:\/\/stackoverflow.com\/questions\/36796183\/why-is-the-maximum-port-range-65535-in-the-tcp-ip-suite\">use 16 bits<\/a>, so they span 0..65535. A proxy URL must carry a host and a port. Reject proxy strings that lack either field.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">require &#8220;net\/http&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">require &#8220;uri&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">require &#8220;openssl&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">require &#8220;zlib&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">require &#8220;stringio&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">def build_http(target_uri, proxy_uri, timeout)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">if proxy_uri<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http = Net::HTTP::Proxy(proxy_uri.host, proxy_uri.port, proxy_uri.user, proxy_uri.password)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">.new(target_uri.host, target_uri.port)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">else<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http = Net::HTTP.new(target_uri.host, target_uri.port)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http.open_timeout = timeout<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http.read_timeout = timeout<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http.use_ssl = (target_uri.scheme == &#8220;https&#8221;)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http.min_version = OpenSSL::SSL::TLS1_2_VERSION if http.use_ssl?<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Set a TLS floor. Many targets drop TLS 1.0 and 1.1. Keep the rule close to the client so it does not drift per call site.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Rotate proxies but keep session stickiness<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Proxy rotation helps when you hit per-IP caps. It also breaks flows that bind a cookie jar to one exit IP. Add a session key, then pin that key to one proxy for a short window.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">For app-style flows, teams often pair proxy pinning with device-like sessions. You can map that work to&nbsp;<a href=\"https:\/\/byteful.com\/blog\/geelark-proxy-integration\">GeeLark: Proxy Setup for the Antidetect Cloud Phone<\/a>.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">class ProxyPool<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">def initialize(proxies, ttl: 120)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">@proxies = proxies.map { |s| URI(s) }<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">@ttl = ttl<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">@by_key = {}<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">def for_key(key, now: Time.now)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">entry = @by_key[key]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">if entry &amp;amp;&amp;amp; (now &#8211; entry[:at]) &amp;lt; @ttl<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">return entry[:proxy]<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">proxy = @proxies.sample<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">@by_key[key] = { proxy: proxy, at: now }<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">proxy<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This pool stays small on purpose. It avoids threads, locks, and complex weight math. Build those only when you measure a need.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Handle 429, 403, and gzip with tight rules<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">HTTP status codes run from 100 to 599. <a href=\"https:\/\/http.dev\/429\">Treat 429 as a hard signa<\/a>l to slow down. Treat 403 as a signal to change headers or proxy, not to spam retries.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">require &#8220;logger&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">def fetch(url, proxy: nil, headers: {}, timeout: 15, logger: Logger.new($stdout))<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">target_uri = URI(url)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">proxy_uri = proxy ? URI(proxy) : nil<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">http = build_http(target_uri, proxy_uri, timeout)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">req = Net::HTTP::Get.new(target_uri)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">req[&#8220;Accept&#8221;] = &#8220;*\/*&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">req[&#8220;Accept-Encoding&#8221;] = &#8220;gzip&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">headers.each { |k, v| req[k] = v }<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">res = http.request(req)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">body = res.body || &#8220;&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">if res[&#8220;Content-Encoding&#8221;] == &#8220;gzip&#8221;<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">gz = Zlib::GzipReader.new(StringIO.new(body))<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">body = gz.read<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">FetchResult.new(code: res.code.to_i, headers: res.to_hash, body: body)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">rescue =&amp;gt; e<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">logger.warn(&#8220;fetch error url=#{url} proxy=#{proxy_uri&amp;amp;.host}: #{e.class}: #{e.message}&#8221;)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">raise FetchError.new(&#8220;fetch failed&#8221;, cause: e)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">end<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Do not auto-retry every error. Retry only on timeouts and 429, and cap tries. Keep backoff simple and deterministic so you can replay runs.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Log for audits and keep compliance close to code<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Put the target host, proxy host, status code, and byte count in each log line. Ruby\u2019s Logger fits this need and keeps output stable across runs. Store raw responses only when a rule or contract requires it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Respect robots rules where they apply to your use case. Honor site terms, and scope data to the need. Keep a deny list of hosts, paths, and query keys that may carry personal data.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Make the fetch layer the single choke point. Teams fix risk faster when one method gates headers, proxy use, and rate rules. Ruby\u2019s standard lib gives enough to ship that layer with few deps.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Scrape jobs fail more from ops drift than from parse bugs. Proxies change, targets add rate caps, and TLS rules shift. Ruby ships solid building blocks for a tight fetch layer. This note defines a small API that fits Ruby core and standard lib style. It keeps method shapes clear, favors explicit args, and works [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":1161,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-1158","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ruby-tips"],"blocksy_meta":{"styles_descriptor":{"styles":{"desktop":"","tablet":"","mobile":""},"google_fonts":[],"version":7}},"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v28.3 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs - Ruby-Doc.org<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs - Ruby-Doc.org\" \/>\n<meta property=\"og:description\" content=\"Scrape jobs fail more from ops drift than from parse bugs. Proxies change, targets add rate caps, and TLS rules shift. Ruby ships solid building blocks for a tight fetch layer. This note defines a small API that fits Ruby core and standard lib style. It keeps method shapes clear, favors explicit args, and works [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/\" \/>\n<meta property=\"og:site_name\" content=\"Ruby-Doc.org\" \/>\n<meta property=\"article:published_time\" content=\"2026-08-25T08:47:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2026-08-25T08:51:04+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2026\/08\/Proxy-Aware-Web-Fetch-in-Ruby.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1672\" \/>\n\t<meta property=\"og:image:height\" content=\"941\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Niall Holloway\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Niall Holloway\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/\"},\"author\":{\"name\":\"Niall Holloway\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#\\\/schema\\\/person\\\/5d70e42a5678a53b2bad87ba3dfa9613\"},\"headline\":\"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs\",\"datePublished\":\"2026-08-25T08:47:00+00:00\",\"dateModified\":\"2026-08-25T08:51:04+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/\"},\"wordCount\":776,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/Proxy-Aware-Web-Fetch-in-Ruby.png\",\"articleSection\":[\"Ruby tips\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/\",\"name\":\"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs - Ruby-Doc.org\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/Proxy-Aware-Web-Fetch-in-Ruby.png\",\"datePublished\":\"2026-08-25T08:47:00+00:00\",\"dateModified\":\"2026-08-25T08:51:04+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#primaryimage\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/Proxy-Aware-Web-Fetch-in-Ruby.png\",\"contentUrl\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/wp-content\\\/uploads\\\/2026\\\/08\\\/Proxy-Aware-Web-Fetch-in-Ruby.png\",\"width\":1672,\"height\":941},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/\",\"name\":\"Ruby-Doc.org\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#organization\",\"name\":\"Ruby-Doc.org\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/07\\\/Ruby-Doc.org_logo_cropped.png\",\"contentUrl\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/wp-content\\\/uploads\\\/2025\\\/07\\\/Ruby-Doc.org_logo_cropped.png\",\"width\":909,\"height\":833,\"caption\":\"Ruby-Doc.org\"},\"image\":{\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/#\\\/schema\\\/person\\\/5d70e42a5678a53b2bad87ba3dfa9613\",\"name\":\"Niall Holloway\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/874bddd2b5926ceb0de4370952d85de984fa9a0a9d2a78ffb035aace075a7da8?s=96&d=mm&r=g\",\"url\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/874bddd2b5926ceb0de4370952d85de984fa9a0a9d2a78ffb035aace075a7da8?s=96&d=mm&r=g\",\"contentUrl\":\"https:\\\/\\\/secure.gravatar.com\\\/avatar\\\/874bddd2b5926ceb0de4370952d85de984fa9a0a9d2a78ffb035aace075a7da8?s=96&d=mm&r=g\",\"caption\":\"Niall Holloway\"},\"url\":\"https:\\\/\\\/ruby-doc.org\\\/blog\\\/author\\\/rubyadmin\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs - Ruby-Doc.org","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/","og_locale":"en_US","og_type":"article","og_title":"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs - Ruby-Doc.org","og_description":"Scrape jobs fail more from ops drift than from parse bugs. Proxies change, targets add rate caps, and TLS rules shift. Ruby ships solid building blocks for a tight fetch layer. This note defines a small API that fits Ruby core and standard lib style. It keeps method shapes clear, favors explicit args, and works [&hellip;]","og_url":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/","og_site_name":"Ruby-Doc.org","article_published_time":"2026-08-25T08:47:00+00:00","article_modified_time":"2026-08-25T08:51:04+00:00","og_image":[{"width":1672,"height":941,"url":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2026\/08\/Proxy-Aware-Web-Fetch-in-Ruby.png","type":"image\/png"}],"author":"Niall Holloway","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Niall Holloway","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#article","isPartOf":{"@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/"},"author":{"name":"Niall Holloway","@id":"https:\/\/ruby-doc.org\/blog\/#\/schema\/person\/5d70e42a5678a53b2bad87ba3dfa9613"},"headline":"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs","datePublished":"2026-08-25T08:47:00+00:00","dateModified":"2026-08-25T08:51:04+00:00","mainEntityOfPage":{"@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/"},"wordCount":776,"commentCount":0,"publisher":{"@id":"https:\/\/ruby-doc.org\/blog\/#organization"},"image":{"@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#primaryimage"},"thumbnailUrl":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2026\/08\/Proxy-Aware-Web-Fetch-in-Ruby.png","articleSection":["Ruby tips"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/","url":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/","name":"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs - Ruby-Doc.org","isPartOf":{"@id":"https:\/\/ruby-doc.org\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#primaryimage"},"image":{"@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#primaryimage"},"thumbnailUrl":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2026\/08\/Proxy-Aware-Web-Fetch-in-Ruby.png","datePublished":"2026-08-25T08:47:00+00:00","dateModified":"2026-08-25T08:51:04+00:00","breadcrumb":{"@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#primaryimage","url":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2026\/08\/Proxy-Aware-Web-Fetch-in-Ruby.png","contentUrl":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2026\/08\/Proxy-Aware-Web-Fetch-in-Ruby.png","width":1672,"height":941},{"@type":"BreadcrumbList","@id":"https:\/\/ruby-doc.org\/blog\/proxy-aware-web-fetch-in-ruby-a-small-stable-api-for-scrape-jobs\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ruby-doc.org\/blog\/"},{"@type":"ListItem","position":2,"name":"Proxy-Aware Web Fetch in Ruby: A Small, Stable API for Scrape Jobs"}]},{"@type":"WebSite","@id":"https:\/\/ruby-doc.org\/blog\/#website","url":"https:\/\/ruby-doc.org\/blog\/","name":"Ruby-Doc.org","description":"","publisher":{"@id":"https:\/\/ruby-doc.org\/blog\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ruby-doc.org\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ruby-doc.org\/blog\/#organization","name":"Ruby-Doc.org","url":"https:\/\/ruby-doc.org\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ruby-doc.org\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2025\/07\/Ruby-Doc.org_logo_cropped.png","contentUrl":"https:\/\/ruby-doc.org\/blog\/wp-content\/uploads\/2025\/07\/Ruby-Doc.org_logo_cropped.png","width":909,"height":833,"caption":"Ruby-Doc.org"},"image":{"@id":"https:\/\/ruby-doc.org\/blog\/#\/schema\/logo\/image\/"}},{"@type":"Person","@id":"https:\/\/ruby-doc.org\/blog\/#\/schema\/person\/5d70e42a5678a53b2bad87ba3dfa9613","name":"Niall Holloway","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/874bddd2b5926ceb0de4370952d85de984fa9a0a9d2a78ffb035aace075a7da8?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/874bddd2b5926ceb0de4370952d85de984fa9a0a9d2a78ffb035aace075a7da8?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/874bddd2b5926ceb0de4370952d85de984fa9a0a9d2a78ffb035aace075a7da8?s=96&d=mm&r=g","caption":"Niall Holloway"},"url":"https:\/\/ruby-doc.org\/blog\/author\/rubyadmin\/"}]}},"_links":{"self":[{"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/posts\/1158","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/comments?post=1158"}],"version-history":[{"count":2,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/posts\/1158\/revisions"}],"predecessor-version":[{"id":1160,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/posts\/1158\/revisions\/1160"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/media\/1161"}],"wp:attachment":[{"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/media?parent=1158"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/categories?post=1158"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ruby-doc.org\/blog\/wp-json\/wp\/v2\/tags?post=1158"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}