The #1 Notebook Collaboration Tool

Learn more

Introducing Treon: Test framework for Jupyter Notebooks

2 minute read

While I love the interactivity of Jupyter notebooks, they lack an awful lot of good software engineering practices (listen to Joel Grus entertaining talk for a wholesale list). My pet peeve is testing. You might ask why is testing even needed for data science, fair question. If you’re doing anything more than a throwaway experiment you need at least some sanity tests & here’s why

  • If you or another soul were to ever open & execute that notebook again it should at least execute without errors (imagine the anguish seeing NameError: name 'pd_gradient' not defined)
  • The notebook should produce reasonably accurate outputs for whatever it is that you were trying to do

If you don’t test the notebook in a fresh kernel there is no gurantee that it’ll even run at all. The best time to fix any issues is right after you’ve created the notebook (and not 6 months down the line when you’re struggling to rerun the analysis).

This is why I built treon, an open source test framework for Jupyter notebooks.

  • Treon runs notebook top to bottom and flags execution errors if any
  • Treon runs unittest & doctest present in your notebook code cells

Installation

pip install treon

Usage

If you don’t specify PATH it’ll recursively search for all notebooks starting from current working directory.

$ treon
Executing treon version 0.1.0
Recursively scanning /workspace/treon/tmp/docs/site/ru/guide for Notebooks...

-----------------------------------------------------------------------
Collected following Notebooks for testing
-----------------------------------------------------------------------
/workspace/treon/tmp/docs/site/ru/guide/keras.ipynb
/workspace/treon/tmp/docs/site/ru/guide/eager.ipynb
-----------------------------------------------------------------------

Triggered test for /workspace/treon/tmp/docs/site/ru/guide/keras.ipynb
Triggered test for /workspace/treon/tmp/docs/site/ru/guide/eager.ipynb

test_sum (__main__.TestNotebook) ...
ok
test_sum (__main__.TestNotebook2) ...
ok
test_sum (__main__.TestNotebook3) ...
ok

----------------------------------------------------------------------
Ran 3 tests in 0.004s

OK

-----------------------------------------------------------------------
TEST RESULT
-----------------------------------------------------------------------
/workspace/treon/tmp/docs/site/ru/guide/keras.ipynb       -- PASSED
/workspace/treon/tmp/docs/site/ru/guide/eager.ipynb       -- PASSED
-----------------------------------------------------------------------
2 succeeded, 0 failed, out of 2 notebooks tested.
-----------------------------------------------------------------------

unitttest example

You just need to add tests as shown below & treon would execute them and report the result on the console. See this for more details on how to write unittest.

doctest example

You just need to add tests as shown below & treon would execute them and report the result on the console. See this for more details on how to write doctest.

Why should you use treon?

  • Start testing notebooks without writing a single line of test code
  • Executes every Notebook in a fresh kernel to avoid hidden state problems
  • Multithreaded execution for quickly testing a set of notebooks
  • Primarily a command line tool that can be used easily in any Continuous Integration (CI) system
  • Soon to be part of ReviewNB’s CI system that automatically runs treon everytime you push notebook changes to GitHub

That’s all folks! Our aim at ReviewNB is to make notebooks a first class entity in data science & ML teams. Hope treon helps you use notebooks more effectively.

You May Also Enjoy