class IO

Practical examples and pitfalls for IO

IO examples: process a stream and distinguish end of input

Practical notes by Ruby-Doc.org

Read only what the count needs

To process each record on a line-by-line basis, you do not need to build another list of all records if the results of reading will simply be some kind of count (e.g., number of "ready" records) or a small summary. The first example illustrates counting ready records while the handle is owned by the outer block.

Example 1
require "tempfile"
Tempfile.create("ruby-doc-example-") do |file|
  file.write("ready\nwaiting\nready\n")
  file.rewind
  count = 0
  file.each_line { |line| count += 1 if line.chomp == "ready" }
  p count
end
Expected output
2

You can control how many parts of the file are kept in memory during line-by-line processing; however, individual lines can still be large. If there is no reasonable limit on the size of input records, take that into consideration when designing both your format and reader. Streaming describes consumption; an individual record can still require substantial memory.

Choose an end-of-input behavior the caller can use

The related read methods may behave differently upon detecting end-of-input. For example, a reader designed to expect a line to always be available can use readline and handle EOFError if the required line is absent. On the other hand, a loop which considers exhaustion as merely normal termination can use gets, which returns nil at end of input.

Example 2
require "stringio"
stream = StringIO.new("ready\n")
p stream.readline
p stream.gets
begin
  stream.readline
rescue EOFError
  puts "End of input"
end
Expected output
"ready\n"
nil
End of input

Ensure there is a distinction between an empty line and end-of-input. A single line composed of just its terminating character(s) is still data that is provided by the reader. Determine whether your record format allows for empty lines prior to discarding them.

Flushing a buffer does not finish an exchange

Writing bytes, flushing buffers, closing handles and completing your application-layer exchange are distinct events. Success of a local write does not indicate that a remote peer has successfully processed your message. Instead, define your own acknowledgement protocol and rely upon those events rather than the existence of an IO object.

When creating test cases for parsers based upon streams, consider including empty input, a last line without a newline character and an ordinary line-ending character. Include encoding choices explicitly at the time that you create your stream. It should not be up to the parser to attempt to decide whether it is receiving characters or raw/uninterpreted binary bytes.

API reference: IO API reference

Related: File ¡ String

An instance of class IO (commonly called a stream) represents an input/output stream in the underlying operating system. Class IO is the basis for input and output in Ruby.

Class File is the only class in the Ruby core that is a subclass of IO. Some classes in the Ruby standard library are also subclasses of IO; these include TCPSocket and UDPSocket.

The global constant ARGF (also accessible as $<) provides an IO-like stream that allows access to all file paths found in ARGV (or found in STDIN if ARGV is empty). ARGF is not itself a subclass of IO.

Class StringIO provides an IO-like stream that handles a String. StringIO is not itself a subclass of IO.

Important objects based on IO include:

An instance of IO may be created using:

Like a File stream, an IO stream has:

And like other IO streams, it has:

Extension io/console

Extension io/console provides numerous methods for interacting with the console; requiring it adds numerous methods to class IO.

Example Files

Many examples here use these variables:

# English text with newlines.
text = <<~EOT
  First line
  Second line

  Fourth line
  Fifth line
EOT

# Russian text.
russian = "\u{442 435 441 442}" # => "тест"

# Binary data.
data = "\u9990\u9991\u9992\u9993\u9994"

# Text file.
File.write('t.txt', text)

# File with Russian text.
File.write('t.rus', russian)

# File with binary data.
f = File.new('t.dat', 'wb:UTF-16')
f.write(data)
f.close

Open Options

A number of IO methods accept optional keyword arguments that determine how a new stream is to be opened:

Also available are the options offered in String#encode, which may control conversion between external and internal encoding.

Basic IO

You can perform basic stream IO with these methods, which typically operate on multi-byte strings:

Position

An IO stream has a nonnegative integer position, which is the byte offset at which the next read or write is to occur. A new stream has position zero (and line number zero); method rewind resets the position (and line number) to zero.

These methods discard buffers and the Encoding::Converter instances used for that IO.

The relevant methods:

Open and Closed Streams

A new IO stream may be open for reading, open for writing, or both.

A stream is automatically closed when claimed by the garbage collector.

Attempted reading or writing on a closed stream raises an exception.

The relevant methods:

End-of-Stream

You can query whether a stream is positioned at its end:

You can reposition to end-of-stream by using method IO#seek:

f = File.new('t.txt')
f.eof? # => false
f.seek(0, :END)
f.eof? # => true
f.close

Or by reading all stream content (which is slower than using IO#seek):

f.rewind
f.eof? # => false
f.read # => "First line\nSecond line\n\nFourth line\nFifth line\n"
f.eof? # => true

Line IO

Class IO supports line-oriented input and output

Line Input

Class IO supports line-oriented input for files and IO streams

File Line Input

You can read lines from a file using these methods:

For each of these methods:

Stream Line Input

You can read lines from an IO stream using these methods:

For each of these methods:

Line Separator

Each of the line input methods uses a line separator: the string that determines what is considered a line; it is sometimes called the input record separator.

The default line separator is taken from global variable $/, whose initial value is "\n".

Generally, the line to be read next is all data from the current position to the next line separator (but see Special Line Separator Values):

f = File.new('t.txt')
# Method gets with no sep argument returns the next line, according to $/.
f.gets # => "First line\n"
f.gets # => "Second line\n"
f.gets # => "\n"
f.gets # => "Fourth line\n"
f.gets # => "Fifth line\n"
f.close

You can use a different line separator by passing argument sep:

f = File.new('t.txt')
f.gets('l')   # => "First l"
f.gets('li')  # => "ine\nSecond li"
f.gets('lin') # => "ne\n\nFourth lin"
f.gets        # => "e\n"
f.close

Or by setting global variable $/:

f = File.new('t.txt')
$/ = 'l'
f.gets # => "First l"
f.gets # => "ine\nSecond l"
f.gets # => "ine\n\nFourth l"
f.close
Special Line Separator Values

Each of the line input methods accepts two special values for parameter sep:

Line Limit

Each of the line input methods uses an integer line limit, which restricts the number of bytes that may be returned. (A multi-byte character will not be split, and so a returned line may be slightly longer than the limit).

The default limit value is -1; any negative limit value means that there is no limit.

If there is no limit, the line is determined only by sep.

# Text with 1-byte characters.
File.open('t.txt') {|f| f.gets(1) }  # => "F"
File.open('t.txt') {|f| f.gets(2) }  # => "Fi"
File.open('t.txt') {|f| f.gets(3) }  # => "Fir"
File.open('t.txt') {|f| f.gets(4) }  # => "Firs"
# No more than one line.
File.open('t.txt') {|f| f.gets(10) } # => "First line"
File.open('t.txt') {|f| f.gets(11) } # => "First line\n"
File.open('t.txt') {|f| f.gets(12) } # => "First line\n"

# Text with 2-byte characters, which will not be split.
File.open('t.rus') {|f| f.gets(1).size } # => 1
File.open('t.rus') {|f| f.gets(2).size } # => 1
File.open('t.rus') {|f| f.gets(3).size } # => 2
File.open('t.rus') {|f| f.gets(4).size } # => 2
Line Separator and Line Limit

With arguments sep and limit given, combines the two behaviors:

Example:

File.open('t.txt') {|f| f.gets('li', 20) } # => "First li"
File.open('t.txt') {|f| f.gets('li', 2) }  # => "Fi"
Line Number

A readable IO stream has a non-negative integer line number:

Unless modified by a call to method IO#lineno=, the line number is the number of lines read by certain line-oriented methods, according to the effective line separator:

A new stream is initially has line number zero (and position zero); method rewind resets the line number (and position) to zero:

f = File.new('t.txt')
f.lineno # => 0
f.gets   # => "First line\n"
f.lineno # => 1
f.rewind
f.lineno # => 0
f.close

Reading lines from a stream usually changes its line number:

f = File.new('t.txt', 'r')
f.lineno   # => 0
f.readline # => "This is line one.\n"
f.lineno   # => 1
f.readline # => "This is the second line.\n"
f.lineno   # => 2
f.readline # => "Here's the third line.\n"
f.lineno   # => 3
f.eof?     # => true
f.close

Iterating over lines in a stream usually changes its line number:

File.open('t.txt') do |f|
  f.each_line do |line|
    p "position=#{f.pos} eof?=#{f.eof?} lineno=#{f.lineno}"
  end
end

Output:

"position=11 eof?=false lineno=1"
"position=23 eof?=false lineno=2"
"position=24 eof?=false lineno=3"
"position=36 eof?=false lineno=4"
"position=47 eof?=true lineno=5"

Unlike the stream’s position, the line number does not affect where the next read or write will occur:

f = File.new('t.txt')
f.lineno = 1000
f.lineno # => 1000
f.gets   # => "First line\n"
f.lineno # => 1001
f.close

Associated with the line number is the global variable $.:

Line Output

You can write to an IO stream line-by-line using this method:

Character IO

You can process an IO stream character-by-character using these methods:

Byte IO

You can process an IO stream byte-by-byte using these methods:

Codepoint IO

You can process an IO stream codepoint-by-codepoint:

What’s Here

First, what’s elsewhere. Class IO:

Here, class IO provides methods that are useful for:

Creating

Reading

Writing

Positioning

Iterating

Settings

Querying

Buffering

Low-Level Access

Other