Tutorial: Extending Tflon

Overview

In this section we demonstrate how to add additional functionality to tflon. For now there’s also just miscellaneous tips that haven’t found a home yet

Tensorflow Basics

It’s a good idea to look at https://tensorflow.org as a reference:

Roughly speaking, Tensorflow represents the network as a graph: collection of neurons and the connections between neurons. This graph of modular pieces can then be distributed across multiple computational units and coordinated in order to optimize training and applying the network.

Pitfall

By the nature of Tensorflow which maintains a stateful representation of the network being trained. In order to instantiate another network, we have to reset the system with

tflon.system.reset()

This will automatically reset the tensorflow stored network. So, that you can define and train another model within the same python file.

Code Structure

Before we add anything to tflon, let’s understand how the code base is organized. Here is a control flow diagram for tflon execution:

First off adding Module should be done by implementing the tflon.toolkit.Module interface

from tflon.toolkit import Module

In particular, there are three main functions in Module that are useful to override: - build: called when the module is initialized (where static variables are populated)

• call: called when the network is constructed (gets called )
• show_data: gets called before data is fed into the module (useful for preprocessing, eg WindowInput)
• featurize: reshapes the data per batch

build and call have no default so any class that implements the module interface must have these two functions defined.

show_data and featurize are called when MyModel.fit is called

tflon/toolkit/modules.py has a great example of implementing custom modules

Hint: Since __getattr__ in Module searches the Tower instance, effectively all the attributes of Tower can be thought of as part of Module.

Example with Highway

We use Highway as an example of how to add a module. The source code is in tflon/toolkit/module.py

See a description of the algorithm here: https://arxiv.org/abs/1505.00387.

class Highway(Module):

    def build(self, block):
        self._gate = Dense(self.input_shapes[0][-1],
            activation=tf.nn.sigmoid, bias_initializer=tf.constant_initializer(-1.))
        self._block = block

    def call(self, inputs):
        hidden = self._block(inputs)
        gate = self._gate(inputs)
        return hidden * gate + inputs * (1-gate)

Incidentally, we can just use the default show_data and featurize

Crash Course on @property

Highly recommend looking at this description if you’re not familiar with @property in python

https://www.programiz.com/python-programming/property

Key Takeaways - ‘_var’ is a convention and python doesn’t treat it any different than any other variable - ‘@’ is a decorator so this

@property
def fn():
    return

becomes

def fn():
    return
fn = property(fn)

• fn = property(fn) takes fn and reassigns fn to be have like a private variable
  • defines a _fn as well as implicitly defines set and get functions -in partictular, the body of fn becomes the get function
  • so now self._fn contains the implicit data value
  • but self.fn will apply the body of fn