Programmer's guide to python

6.9 Generators and Coroutines

>Generators are used to generate streams of data. Instead of loading all the data at once (example like list object does) they "lazy load" the data, which means they return a value only when the next() function is called upon them. The next() function is used to fetch the next element from the generator object. Now the fetching can be from databases or generated on the fly.
>Generators are iterators (we'll learn more about them in next Chapter) objects, so they can be iterated using loops. A generator object is created using a function with one or more yield statements in it. yield allows generators to be iterated with/without defining loops in their function.
>yield helps to save the state (or maintains current index of iteration) of a generator object, this allows generators to be interrupted and resumed throughout the program's execution. Once a generators data is exhausted it stops returning values and raises StopIteration, at this point it needs to be initialized again.
>For longer iteration (larger/infinite length of data handling) generators are preferred because they are memory efficient, in a sense they can be utilized to generate/load data when required. This helps in avoiding the machine to run out of memory. Lastly, generators can also be created using comprehensions, using the rounded brackets.
>Creating a simple generator.

## Example 1: Create a generator function which returns square of each values
def square_generator(*args):
  for a in args:
      yield a ** 2
      # Notice: Using 'yield' instead of 'return' makes this function "lazy"
      # and hence a generator

generator = square_generator(2, 3, 4)
print(type(generator))  # <class 'generator'>

# fetch the value using the 'next()' function
print(next(generator))  # 4
# whenever a value is returned the generator is paused at the 'yield' statement
# now as the 'next()' is called again, the 'for' loop will be resumed and so on
print(next(generator))  # 9
print(next(generator))  # 16
# once the generator is exhausted, 'StopIteration' is raised
print(next(generator))  # StopIteration


## Creating the above generator using comprehensions
generator = (a ** 2 for a in [2, 3, 4, 5])
print(type(generator))  # <class 'generator'>
print(hasattr(generator, "__next__"))  # True
# fetch the first value
print(next(generator))  # 4


## Iterating a generator
for a in generator:
  print(a)  # 9 16 25
  # Notice: Generator resumed after first value

# creating a new generator
generator = (x ** 2 for x in [2, 3, 4, 5])
for v in generator:
  print(v)  # 4 9 16 25
  # Notice: We didn't call 'next()' first, so all values are iterated properly

# however now it is exhausted, so will raise error
print(next(generator))  # StopIteration

>The pause and play of generators.

def some_generator():
  print("Hello 1")
  yield 10
  print("Hello 2")
  yield 20
  print("Hello 3")
  yield 30


## Creating a generator
my_generator = some_generator()
# calling the 'next()' on generator will execute till first 'yield' statement
print(next(my_generator))  # Hello 1 \
                         # 10
# now calling the 'next()' again will resume from line 3 till next 'yield'
# statement which is 'yield 20'
print(next(my_generator))  # Hello 2 \
                         # 20
# and so on..
# This is how the generator saves its state at yield statement,
# allowing the object to be interrupted and resumed anytime

>Iterating generators that don't have loops in them.

def my_generator():
  yield 1
  yield 2
  yield 3

generator = my_generator()


## Now iterating the generator
for a in generator:
  print(a)  # 1 2 3
# behind the scenes the 'for' loop is actually calling the 'next()' function
# on each iteration, which is why in our 'square_generator' example the
# 'for' loop was able to continue from the interruption


## yield from: Is used to 'yield' from a sequence
# or even a generator (it'll be called a sub generator)
# the above function can be coded as following
def my_generator():
  yield from [1, 2, 3]

for a in my_generator():
  print(a)  # 1 2 3

>Coroutines are similar to generators, but instead of generating data they are used for consuming the data. Similar to a generator, a coroutine also has yield in its function. The difference is that the yield is assigned to a variable, it is used to pass some value into the function. So a generator becomes a coroutine if its yield is assigned to a variable.
>Coroutines can also maintain their state just like a generator, allowing a constant flow of input data. There are three main methods associated with the coroutine objects, they are send(), close() and throw(). So, send() is used to send value into a coroutine, close() is used to terminate the generator/coroutine execution and throw() is used to raise an Exception into a generator function.
>Coroutines are easy to define and can be combined with other generator/coroutine, this helps in building pipelines and in other asynchronous workflows.
>Creating a coroutine.

## Define a coroutine function
def my_coroutine():
  print("Coroutine is activated..")
  # Note: The 'yield' is assigned to a variable
  val = yield
  print(f"{val} received")

# Note: A coroutine is activated only after reaching the first 'yield' statement
# to activate you need to call 'next()' on its object first
coroutine = my_coroutine()
next(coroutine)  # Coroutine is activated..
# And now you can start sending in values
# Note: If you call 'next()' at this point 'None' will be send
coroutine.send(42)  # 42 received
# At this point our coroutine has started looking for the next 'yield' statement
# if it doesn't find any, the coroutine will terminate itself,
# raising 'StopIteration',
# Note: Comment line number 14 to continue further


## To stop this from happening, create a infinite loop and call
# 'close()' method when done
def my_coroutine():
  print("Coroutine is activated..")
  while True:
      val = yield
      print(f"{val} received")

# create and activate 'my_coroutine'
coroutine = my_coroutine()
next(coroutine)  # Coroutine is activated..
# now sending values
coroutine.send(42)  # 42 received
coroutine.send(34)  # 34 received
coroutine.send(32)  # 32 received
# when done call the 'close()' to terminate the coroutine
coroutine.close()

>Another example of coroutine.

## Example: Percent calculator using coroutine
# Note: When pairing 'yield' with a expression it is recommended to
# use parentheses like this '(yield)'
def percent_coroutine(total):
  while True:
      result = (yield) / total * 100
      # limiting float number to single decimal
      print(f"Your Percentage are {round(result, 1)}")

percent_calc = percent_coroutine(420)
next(percent_calc)

percent_calc.send(250)  # Your Percentage are 59.5
percent_calc.send(356)  # Your Percentage are 84.8
percent_calc.send(155)  # Your Percentage are 36.9
percent_calc.close()

>Using generator and coroutine together.

## Example: Create a fibonacci generator and a coroutine to filter
# the even numbers and return it
def fib_gen(limit=10):
  """
  This is a generator function that generates fibonacci numbers.
  """
  x, y = 0, 1
  for _ in range(limit):
      x, y = y, x + y
      yield x
  # at last send 'None' to coroutine, to signal for stopping
  yield None

def co_fetcher():
  """
  This is a coroutine function that takes numbers and stores
  the even numbers in a list and finally returns the list.
  """
  even_fibs = []
  while True:
      num = yield
      # stopping condition for the coroutine
      if not num:
          break
      if num % 2 == 0:
          even_fibs.append(num)
  return even_fibs


## Create a generator
fib = fib_gen()


## Create and activate the coroutine
fetcher = co_fetcher()
next(fetcher)

# Note: As the coroutine is terminated it raises 'StopIteration',
# so the final returned value should be inside the 'value' attribute
# of the exception
while True:
  try:
      # send values generated from generator to coroutine
      fetcher.send(next(fib))
  except StopIteration as e:
      print(f"Your even fibonacci are {e.value}")  # Your even \
                                      # fibonacci are [2, 8, 34]
      # make sure to stop this loop too
      break