Written by James BrittTechnically reviewed by Jim Freeze · Reviewed
A scraper that worked on Friday and fails on Monday may have nothing wrong with its parser. The page might now be a login form, the proxy credentials could have expired, or the server could be asking the client to slow down. Those failures become harder to diagnose when every caller handles the network request differently.
For a Ruby batch job, a small shared fetch method is often enough to bring that behaviour into one place. This example uses Net::HTTP and URI, accepts an optional HTTP proxy, and returns the information the job needs before it starts parsing.
Give the job a useful response
The method returns a FetchResult containing the status code, headers and body. Receiving a 403 or 429 still counts as receiving a response; the caller gets to decide what happens next. A connection failure takes the exception path instead.
That leaves a fairly simple division of work. The fetcher deals with the HTTP request. The job knows whether a particular response is useful, whether a retry is justified and how long the whole import is allowed to run.
Save this as fetch.rb. It uses APIs documented for Ruby 3.4, with the required standard-library gems installed.
require "net/http"
require "uri"
require "openssl"
require "zlib"
require "logger"
require "json"
FetchResult = Struct.new(:code, :headers, :body, keyword_init: true)
class FetchError < StandardError; end
def fetch(url, proxy: nil, headers: {}, timeout: 15, logger: nil)
target = URI.parse(url)
unless target.is_a?(URI::HTTP) && target.hostname &&
!target.userinfo && (1..65_535).cover?(target.port)
raise ArgumentError, "expected an HTTP(S) URL without credentials"
end
seconds = Float(timeout)
unless seconds.finite? && seconds.positive?
raise ArgumentError, "timeout must be positive and finite"
end
gateway = proxy ? URI.parse(proxy) : nil
if gateway
unless gateway.scheme == "http" && gateway.hostname &&
(1..65_535).cover?(gateway.port) &&
["", "/"].include?(gateway.path) &&
!gateway.query && !gateway.fragment
raise ArgumentError, "expected an HTTP proxy URL"
end
decode = ->(value) { value && URI::DEFAULT_PARSER.unescape(value) }
http = Net::HTTP.new(
target.hostname, target.port, gateway.hostname, gateway.port,
decode.call(gateway.user), decode.call(gateway.password)
)
else
http = Net::HTTP.new(target.hostname, target.port, nil)
end
http.open_timeout = seconds
http.read_timeout = seconds
http.write_timeout = seconds
http.max_retries = 0
http.use_ssl = target.scheme == "https"
if http.use_ssl?
http.min_version = OpenSSL::SSL::TLS1_2_VERSION
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.verify_hostname = true
end
request = Net::HTTP::Get.new(target, headers)
context = { host: target.hostname, proxy_host: gateway&.hostname }
begin
response = http.request(request)
rescue Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout,
SocketError, SystemCallError, IOError,
OpenSSL::SSL::SSLError, Net::ProtocolError,
Net::HTTPBadResponse, Net::HTTPHeaderSyntaxError, Zlib::Error => error
logger&.warn(JSON.generate(context.merge(
event: "fetch_error", code: nil, bytes: 0, error: error.class.name
)))
raise FetchError, "HTTP fetch failed", cause: error
end
body = response.body || ""
logger&.info(JSON.generate(context.merge(
event: "fetch", code: response.code.to_i, bytes: body.bytesize
)))
FetchResult.new(code: response.code.to_i, headers: response.to_hash, body: body)
endThere is no custom implementation of cause here. Ruby already supports exception chaining through raise ... cause: error. Configuration errors are also allowed to surface before the request, rather than being caught by a broad rescue StandardError and reported as network trouble.
The rescue clause covers common transport, protocol and decompression failures. Other exceptions can still escape it.
Why the proxy argument matters
The direct connection branch contains an easy detail to miss:
Net::HTTP.new(target.hostname, target.port, nil)That final nil disables proxy discovery from the environment. Leaving it out allows Net::HTTP to pick up http_proxy. This can explain why the same script takes different routes on a laptop and a scheduled worker. Here, a caller passing proxy: nil gets a direct connection.
The other branch accepts an HTTP proxy. For an HTTPS destination, the connection uses a CONNECT tunnel, with TLS protecting traffic to the target. SOCKS and TLS to the proxy itself are outside this example.
Ruby supplies port 80 when an HTTP proxy URL omits its port. A provider using 8080 or another port needs that value included in the URL. Consequently, checking URI#port alone cannot tell you whether the original string specified a port.
Credentials belong in deployment configuration or a secret store. The helper decodes percent-encoded usernames and passwords before passing them to Net::HTTP; the full proxy URL is never included in its log message.
Check what came back
A caller can use the helper like this:
require_relative "fetch"
result = fetch(
"https://example.com/",
proxy: ENV["FETCH_PROXY_URL"],
headers: { "Accept" => "text/html" },
logger: Logger.new($stdout)
)
puts result.code
puts result.headers["content-type"]&.first
puts result.body.bytesizeFor a normal 200 HTML response, those lines print 200, its content type and the body’s size in bytes. The last two values depend on the server. The header hash uses lower-case names and arrays of values.
An HTML parser will quite happily parse an access-denied page. Checking the status and expected content before parsing helps prevent an empty import from looking like a successful run.
The helper returns redirects to the caller, too. It has no automatic redirect following, cookie jar or connection reuse between calls.
What the timeout actually limits
The timeout argument is applied to opening, reading and writing. It does not put a 15-second ceiling on the entire request: a read timeout limits an individual blocking read. A server that keeps sending small chunks may keep the transfer running much longer.
The import therefore needs its own time budget. A short retry after an opening or reading timeout may be reasonable for a GET, but repeated attempts still consume that budget. Certificate failures need investigation; turning off verification would hide the problem.
Setting max_retries = 0 disables the client’s built-in retry mechanism so that the job can own this decision. HTTPS certificate and hostname verification remain enabled, with TLS 1.2 as the minimum protocol version.
Different responses call for different handling:
| Result | What the job should consider |
|---|---|
| Expected 200 response | Check the content, then pass it to the parser. |
| 403 Forbidden | Investigate permissions, credentials and access requirements. |
| 429 Too Many Requests | Respect a usable Retry-After value and reduce the request rate. |
| Opening or reading timeout | Retry only within a bounded attempt and time budget. |
A 429 response may supply Retry-After. That value can be seconds or an HTTP date. When its delay is longer than the worker can wait, rescheduling preserves the server’s requested pause. Without a usable value, bounded backoff with jitter helps keep multiple workers from retrying together.
A 403 response means the server understood the request but refuses to fulfil it. It is not, by itself, evidence that the proxy needs replacing.
Leave ordinary gzip handling to Net::HTTP
The example leaves Accept-Encoding alone. With zlib support available, Net::HTTP normally negotiates supported compression and decompresses the response. Adding a second gzip decoder can mean trying to decompress content that has already been decoded.
A caller that sets Accept-Encoding itself takes responsibility for the changed decoding behaviour. Range requests also need separate treatment.
This matters when reading the logs: body.bytesize measures the returned body. After automatic decompression, it tells you how many decoded bytes the parser receives, rather than how many bytes crossed the network.
Keep related requests on the same session
Some authorised workflows associate a login with an IP address. Choosing a random proxy for every request can disrupt them. The job can instead select a proxy once and pass it through the related calls, while a separate cookie jar holds that session’s cookies.
A stable proxy URL does not always guarantee a stable exit IP. A provider’s rotating endpoint may need its own session identifier. Equally, keeping the same IP does not create a cookie jar in Ruby.
Where a wider workflow includes a cloud phone, Byteful’s GeeLark: Proxy Setup for the Antidetect Cloud Phone covers that product’s configuration. This fetcher remains a separate Ruby HTTP client; it does not reproduce a mobile browser or device session.
If the job grows into a pool of sessions, expiry and concurrency become part of the design. A monotonic clock is suitable for measuring elapsed time, and expired mappings need removing so old session keys do not accumulate indefinitely.
Make the logs useful without storing the response
The helper records the target host, proxy host, status and returned byte count. Its failure log contains an exception class. That is enough to distinguish several common problems without placing passwords, cookies, bodies or full URLs in routine logs.
Query strings can contain tokens or personal information. Even exception messages may include connection details, which is why the example logs the class rather than the message. The underlying exception remains accessible through FetchError#cause for controlled diagnosis.
There are two limits to address before expanding this into a general-purpose service. First, the whole body is read into memory; large or unpredictable responses need streaming with a maximum decoded size. Second, accepting arbitrary user-supplied URLs requires destination restrictions and DNS/IP checks. Scheme validation alone does not prevent server-side request forgery.
For a scheduled collection job, approved destinations, request rates and retention rules should live alongside its configuration. The target’s published access requirements and robots guidance belong in that review. From there, an observed problem can guide the next change, whether that is connection reuse, a shared rate limiter or a response-size limit.