class Enumerator

Practical examples and pitfalls for Enumerator

Enumerator examples: make progress and stopping explicit

Practical notes by Ruby-Doc.org

Use an enumerator when the caller controls the pace

Some tasks are easier to describe as requests for the next item. A small command-line chooser, for example, may advance through available labels one step at a time. Calling each without a block gives the caller an enumerator, and next advances its external iteration position.

Example 1
labels = %w[queued packed].each
p labels.next
p labels.next
begin
  labels.next
rescue StopIteration
  puts "No labels left"
end
Expected output
"queued"
"packed"
No labels left

The exception is part of the end-of-input contract. Handle StopIteration where exhaustion is expected, rather than treating every exception as the end of the list. A parsing failure or failed network request should remain distinguishable from having no items left. For an ordinary pass over all items, using each with a block is usually clearer than managing this position yourself.

Produce values only when the caller asks

An enumerator can also package a producer. The second example describes the start offset of each batch. It yields offsets as they are requested, leaving the caller to choose how many batches to inspect. It does not allocate an infinite array of offsets.

Example 2
offsets = Enumerator.new do |out|
  offset = 0
  loop do
    out << offset
    offset += 25
  end
end
p offsets.take(4)
Expected output
[0, 25, 50, 75]

Treat the batch size as part of the producer's contract. A real helper should reject a zero or negative size, and should explain whether its last batch can be shorter than the others. Producing offsets is only the scheduling part; fetching records at those offsets may have its own consistency rules if the data changes between requests.

Keep position separate from the underlying data

An enumerator is not necessarily a snapshot. If its producer reads a changing collection or an external resource, later values can reflect that source. rewind resets enumeration state and may call the underlying object's rewind method when available; it cannot promise to restore an external system to an earlier state.

Test exhaustion, partial consumption and any intended repeat traversal. If callers should receive a fresh traversal each time, return a new enumerator from a method instead of sharing one advancing instance across requests. That makes ownership of iteration progress much easier to understand.

API reference: Enumerator API reference

Related: Enumerable · Range

A class which allows both internal and external iteration.

An Enumerator can be created by the following methods.

Most methods have two forms: a block form where the contents are evaluated for each item in the enumeration, and a non-block form which returns a new Enumerator wrapping the iteration.

enumerator = %w(one two three).each
puts enumerator.class # => Enumerator

enumerator.each_with_object("foo") do |item, obj|
  puts "#{obj}: #{item}"
end

# foo: one
# foo: two
# foo: three

enum_with_obj = enumerator.each_with_object("foo")
puts enum_with_obj.class # => Enumerator

enum_with_obj.each do |item, obj|
  puts "#{obj}: #{item}"
end

# foo: one
# foo: two
# foo: three

This allows you to chain Enumerators together. For example, you can map a list’s elements to strings containing the index and the element as a string via:

puts %w[foo bar baz].map.with_index { |w, i| "#{i}:#{w}" }
# => ["0:foo", "1:bar", "2:baz"]

External Iteration

An Enumerator can also be used as an external iterator. For example, Enumerator#next returns the next value of the iterator or raises StopIteration if the Enumerator is at the end.

e = [1,2,3].each   # returns an enumerator object.
puts e.next   # => 1
puts e.next   # => 2
puts e.next   # => 3
puts e.next   # raises StopIteration

next, next_values, peek, and peek_values are the only methods which use external iteration (and Array#zip(Enumerable-not-Array) which uses next internally).

These methods do not affect other internal enumeration methods, unless the underlying iteration method itself has side-effect, e.g. IO#each_line.

FrozenError will be raised if these methods are called against a frozen enumerator. Since rewind and feed also change state for external iteration, these methods may raise FrozenError too.

External iteration differs significantly from internal iteration due to using a Fiber:

Concretely:

Thread.current[:fiber_local] = 1
Fiber[:storage_var] = 1
e = Enumerator.new do |y|
  p Thread.current[:fiber_local] # for external iteration: nil, for internal iteration: 1
  p Fiber[:storage_var] # => 1, inherited
  Fiber[:storage_var] += 1
  y << 42
end

p e.next # => 42
p Fiber[:storage_var] # => 1 (it ran in a different Fiber)

e.each { p _1 }
p Fiber[:storage_var] # => 2 (it ran in the same Fiber/"stack" as the current Fiber)

Convert External Iteration to Internal Iteration

You can use an external iterator to implement an internal iterator as follows:

def ext_each(e)
  while true
    begin
      vs = e.next_values
    rescue StopIteration
      return $!.result
    end
    y = yield(*vs)
    e.feed y
  end
end

o = Object.new

def o.each
  puts yield
  puts yield(1)
  puts yield(1, 2)
  3
end

# use o.each as an internal iterator directly.
puts o.each {|*x| puts x; [:b, *x] }
# => [], [:b], [1], [:b, 1], [1, 2], [:b, 1, 2], 3

# convert o.each to an external iterator for
# implementing an internal iterator.
puts ext_each(o.to_enum) {|*x| puts x; [:b, *x] }
# => [], [:b], [1], [:b, 1], [1, 2], [:b, 1, 2], 3