Scrape jobs fail more from ops drift than from parse bugs. Proxies change, targets add rate caps, and TLS rules shift. Ruby ships solid building blocks for a tight fetch layer.
This note defines a small API that fits Ruby core and standard lib style. It keeps method shapes clear, favors explicit args, and works well in batch jobs. Use it as a base for Byteful data pulls where you need repeat runs and clean logs.
Define a narrow fetch contract
Start with one call that returns three fields: status, headers, body. Avoid hidden globals. Route all net errors to one exception type.
Signature
FetchResult = Struct.new(:code, :headers, :body, keyword_init: true)
class FetchError < StandardError
attr_reader :cause
def initialize(msg, cause: nil)
@cause = cause
super(msg)
end
end
# @param url [String]
# @param proxy [String, nil] “http://user:pass@host:port”
# @param headers [Hash{String => String}]
# @param timeout [Numeric]
# @return [FetchResult]
def fetch(url, proxy: nil, headers: {}, timeout: 15)
end
Ruby-doc readers expect small objects and clear args. Use URI for parse and Net::HTTP for I/O. Keep default timeouts short so jobs fail fast.
Wire Net::HTTP with explicit proxy and TLS
TCP ports use 16 bits, so they span 0..65535. A proxy URL must carry a host and a port. Reject proxy strings that lack either field.
require “net/http”
require “uri”
require “openssl”
require “zlib”
require “stringio”
def build_http(target_uri, proxy_uri, timeout)
if proxy_uri
http = Net::HTTP::Proxy(proxy_uri.host, proxy_uri.port, proxy_uri.user, proxy_uri.password)
.new(target_uri.host, target_uri.port)
else
http = Net::HTTP.new(target_uri.host, target_uri.port)
end
http.open_timeout = timeout
http.read_timeout = timeout
http.use_ssl = (target_uri.scheme == “https”)
http.min_version = OpenSSL::SSL::TLS1_2_VERSION if http.use_ssl?
http
end
Set a TLS floor. Many targets drop TLS 1.0 and 1.1. Keep the rule close to the client so it does not drift per call site.
Rotate proxies but keep session stickiness
Proxy rotation helps when you hit per-IP caps. It also breaks flows that bind a cookie jar to one exit IP. Add a session key, then pin that key to one proxy for a short window.
For app-style flows, teams often pair proxy pinning with device-like sessions. You can map that work to GeeLark: Proxy Setup for the Antidetect Cloud Phone.
class ProxyPool
def initialize(proxies, ttl: 120)
@proxies = proxies.map { |s| URI(s) }
@ttl = ttl
@by_key = {}
end
def for_key(key, now: Time.now)
entry = @by_key[key]
if entry && (now – entry[:at]) < @ttl
return entry[:proxy]
end
proxy = @proxies.sample
@by_key[key] = { proxy: proxy, at: now }
proxy
end
end
This pool stays small on purpose. It avoids threads, locks, and complex weight math. Build those only when you measure a need.
Handle 429, 403, and gzip with tight rules
HTTP status codes run from 100 to 599. Treat 429 as a hard signal to slow down. Treat 403 as a signal to change headers or proxy, not to spam retries.
require “logger”
def fetch(url, proxy: nil, headers: {}, timeout: 15, logger: Logger.new($stdout))
target_uri = URI(url)
proxy_uri = proxy ? URI(proxy) : nil
http = build_http(target_uri, proxy_uri, timeout)
req = Net::HTTP::Get.new(target_uri)
req[“Accept”] = “*/*”
req[“Accept-Encoding”] = “gzip”
headers.each { |k, v| req[k] = v }
res = http.request(req)
body = res.body || “”
if res[“Content-Encoding”] == “gzip”
gz = Zlib::GzipReader.new(StringIO.new(body))
body = gz.read
end
FetchResult.new(code: res.code.to_i, headers: res.to_hash, body: body)
rescue => e
logger.warn(“fetch error url=#{url} proxy=#{proxy_uri&.host}: #{e.class}: #{e.message}”)
raise FetchError.new(“fetch failed”, cause: e)
end
Do not auto-retry every error. Retry only on timeouts and 429, and cap tries. Keep backoff simple and deterministic so you can replay runs.
Log for audits and keep compliance close to code
Put the target host, proxy host, status code, and byte count in each log line. Ruby’s Logger fits this need and keeps output stable across runs. Store raw responses only when a rule or contract requires it.
Respect robots rules where they apply to your use case. Honor site terms, and scope data to the need. Keep a deny list of hosts, paths, and query keys that may carry personal data.
Make the fetch layer the single choke point. Teams fix risk faster when one method gates headers, proxy use, and rate rules. Ruby’s standard lib gives enough to ship that layer with few deps.