Lecture 05: A bit about Jupyter

Download on GitHub

1. Workflow and debugging
2. Programming is more than writing code
3. Design patterns
4. Debugging
5. VSCode
6. Modules
7. Git
8. Summary

A few pointers about working in JupyterLab

• Basically, there are 2 kinds of cells: code (press y) and markdown (press m)
• Run a cell: Ctrl + Enter
• Delete a cell: x
• Create a cell above current cell: a
• Create a cell below current cell: b
• Write inline latex in markdown cell, eg. an $\alpha$ : $\alpha$
• Write separate equation line latex in markdown cell: use 2 dollar signs : $$\alpha = \beta$$
• Create titles in markdown cell: # for top level section, ## for second level, etc.

See icon on the left bar for many more commands and hot keys.

Lets quickly look at some variable through the Variable Inspector, also found at the bottom of

1. Workflow and debugging

You will learn how to structure and comment your code and document it for later use. You will learn how to debug your code using print, assert and try/except statements. You will learn how to write modules and run scripts from a terminal in VSCode and how to share your code with others through Git.

2. Programming is more than writing code

You seldom write some code, run it, get the right results, and then never use it again.

• Firstly: You make errors (bugs) when you code.
• Secondly: You need to share your code with colleagues and your future self.

Transparent macro- and microstructure is important:

• For preventing errors.
• For finding errors.
• For making your code interpretable for others and your future-self.

No code is self-explanatory - even though if might seem so when you write it.

Cleaning, commenting and documenting code takes time, but is a crucial aspect of good programming.

In scientific programming, a transparent program structure and good documentation is also a cornerstone in securing replicability.

2.1 Structure

Macro structure (wrt. folders and files):

1. One folder for each project with ALL required files.
2. End goal: 1 file (notebook) to run it all. Very important!
3. Module files (.py): Define functions, classes, etc. Perhaps different modules for different kind of tasks (solving, simulating, plotting).
4. Notebook files (.ipynb): Call functions, classes etc. and explain and present the results.
5. Larger projects: Sub-folders for data, figures, etc. (not relevant now).

Workflow:

1. Notebooks (.ipynb): Work with them in JupyterLab.
2. Modules (.py): Work with them preferably in VSCode (but JupyterLab will also do).

Microstructure:
How do you structure your code and format it like a pro?
Answer is the official PEP8 guideline.
Will tell you all about how to do formatting, commenting and structuring Python code. Makes your code readable to others.

Note: A style guide is about consistency. Consistency with this style guide is important. Consistency within a project is more important. Consistency within one module or function is the most important.

Recommendations:

1. Code layout:
   • Indentation: Four spaces
   • Line length: Max of 79 characters (wrap line + indent properly)
   • Strings: Use single or double quote (be consistent)
   • White space:
     • After comma: x = [1, 2, 3] (not required)
     • Around assignment: x = y
     • After colon: if x == 2: print(x)
     • Around operators with lowest priority in a calculation: c = (a+b) * (a-b) or z = x*x + y*y
2. Naming conventions: Short, but also precise
   • Modules: Lower case with potential underscores (e.g. numecon or num_econ)
   • Classes: Camel case (e.g. ConsumerClass)
   • Variables, functions and methods: Lower case with potential underscores
3. Ordered section comments: Break your code into sections
   • Give each section a name and a place in the ordering
   • Level 1: a, b, c etc.
   • Level 2: i, ii, iii, iv etc.
   • Level 3: o, oo, ooo, oooo etc.
4. Line comments: Small additional hints
   • Again, short and precise
   • Avoid just explaining what the code does (must provide additional information)
5. Docstrings: Should be written for all functions, methods and classes (see how below).

More on names:

1. Name functions after their intended use. Verbs can be handy for such naming. (But its doing many things? Not a good sign, see design patterns below)
2. Help your self in debugging and name variables in a searchable way (unless they are super local). You cannot search for the name i in a bunch of code files.
3. Normally avoid using any special characters.
4. Unused variables and non-public methods should start with a _

Two different perspectives on comments:

1. The comments explain humans what the code does.(~ you'll write the code first, then comments)
2. The code makes the computer do what the comments say. (~ you'll write the comments first, then code)

Example of well formatted code:

Try: Write my_function( and press Shift+Tab

Recommendation: Try to think about which sections and sub-sections you need beforehand. You can even write before you write code!

3. Design patterns

When thinking about how organize your functions and objects, few commandments that will serve you well:

1. DRY: Do not Repeat Yourself. A specific line of code must only appear once in your script. Get rid of code repetitions by looping or create functions for the lines that are being repeated. Code repetition induces bugs when you change code.
2. 1 job: A function has 1 job only. That is, it should only try to accomplish one well-defined task. Sub-tasks within the main task is delegated to other functions.
3. No side effects: A corollary of 1 job. If a function returns $x$, then it should not also produce lasting changes to $y$ if $y$ lives outside the local scope of the function.
4. 1 screen fits all: A handy rule-of-thumb is that the body of code (not including doc strings) in a function should fit into your screen in a readable way. One should not be too religious about this principle, though. But if you have one long function after the other, you have probably violated the commandments above.

More on design patterns:

• You can check out Google's Python style guide to catch a quick glimpse of how they organize their work.

• One of the bibles on design patterns is edited by the famous Uncle Bob. The code examples are not based on Python, but the logic and insights still apply. It's actually fun and not-that-hard to read - and full of wisdom!

4. Debugging

Why is a programming error called a bug?

General advice:

1. Code is always partly a black box: Print and plot results to convince yourself (and others) that your results are sensible.
2. Errors are typically something very very simple, look after that.
3. If Python raises an error first try to locate the line where the error occurs.
4. Your code can often run, but give you unexpected behavior.
5. Include if, print and assert statements to catch errors.

Most of the time spend programming is debugging!! Even when the final code is simple, it can take a lot of trial-and-error to get there.

Assertions: Whenever you know something about your variables (e.g. that they should be positive), you should assert this. If the assertion does not hold Python raises an error.
Exceptions: When code fails, it generates ('raises') an exception.

Task: Make the above assertion fail.

4.1 Example

See official Introduction to errors and exceptions and RealPython on exceptions.

Consider the following code:

Problem: Our result is a complex number. We did not expect that. Why does this problem arise?

Find the error with print:

Solution with an assert:

Solution with if and raise exception:

Note: You could also decide that the function should return e.g. \( -\infty \) when experiencing a complex number.

4.2 Numpy warnings

Here we see an example of an error warning thrown by numpy. Notice the term RuntimeWarning.

Run time is the moment a function is executed by the computer - so a run time error is a kind of error that cannot be detected before the program actually runs.
In the present case: Jupyter/Python simply cannot detect that the -1 in xlist will get in conflict with the log function before the cell is run.
However, you may see that the VS Code Intellisense informs you that perhaps some syntax is wrong or that a variable has not been defined before it is put in use.

You can ignore all warnings:

Better: Decide what the code should do.

4.3 Scope bugs

Global variables are dangerous:

Question: What is the error?

Conclusion: Never use global variables, they can give poisonous side effects. Use a positional or a keyword argument instead.

Useful tool I: The variable inspector.

1. Install: See here
2. Open it: Right-click and choose "Open Variable Inspector"

Useful tool II: The console.

1. Install: Done automatically
2. Open it: Right-click and choose "New Console for Notebook"

4.4 Index bugs

Task: Solve the problem.

5. VSCode

Central benefits of VSCode:

1. Good editor (easy to move across and with-in files)
2. Linting (find errors before you run the code)
3. Run scripts
4. Interactive sessions
5. Integrated git (to share your code online) (see below)
6. Debugging (not today)

Example: We go through this guide together.

6. Modules

Long notebooks can be very hard to read. Code is structured better in modules saved in .py files.

1. Open VSCode
2. Locate the folder with your notebook
3. Create mymodule.py
4. In the notebook: import mymodule
5. All functions in mymodule.py is now avaliable in the notebook with the prefix mymodule.

Important: if you write changes in the code of your own module, eg. mymodule, and if mymodule has already been imported to Jupyter before the changes, then simply running the import mymodule statement again will not import your changes. Python sees that mymodule is already imported, and thus does nothing.

Solution: Use the %load_ext autoreload magic with %autoreload 2. Then your modules are automatically reloaded each time you run a cell. Without the module is never reloaded.

Another solution is to use importlib

Extra: Locating modules

You may wonder: how does Python actually find modules on my computer?
And: what if I made 2 modules with the same name, or created a module with a name that was already present in Anaconda?

Simple answer: when loading modules, Python browses through a predetermined set of folders on your computer, defined by the sys.path.
The first time it encounters a module with the name of the module to be loaded, it brings that one up. If have 2 similarly named modules, the first on the list would therefore be chosen.

You can inspect the order of folders where Python searches for modules on your computer by running the cell below. Notice that your current working directory is second on the list.

7. Git

The purpose of git is to allow you to easily share your code with collaborators and track the changes each of you make.

We go through this guide together.

Note: You will be given repositories named github.com/NumEconCopenhagen/projects-2022-YOURGROUPNAME

Essential Git terms:

• Local your computer.
• Remotes the code on Github and on other computers.
• Branch a branch of code is a separate track or copy of the code base on which you can develop new stuff. There is normally a structure of a main branch that holds the current working version of code and then several testing branches where new stuff is developed. After development, those braches are merged onto the main branch.
• .gitignore a file that contains specifications on which types of files that are not included in process of sending changes back and forth.
• .git there is a hidden folder in all git repositories. This folder includes the diff and head files that contain the whole history of changes to code so far. Delete .git, and your code folders are no longer a working repository. Now it's just regular code.

Essential Git commands:

• Fetch is the process of getting aware of any changes to code outside the local repository on your computer. Does not happen automatically! You are not importing changes by fetching, you just make your local check if anything has happened on the remote repo(s).
• Stage before you can send off your own changes to code, you need to decide which chunks of code specifically to send. Mostly, you will just stage all, that is, send off all changes you have made.
• Commit the process of making your changes available to the remote repo.
• Merge when you let changes to code from remotes get weaved into your own code.
• Push after committing, you order the remote take the changes you made. The remote will not automatically accept the order, if you are not the admin of the remote repo.
• Pull is fetching and merging with the remote.
• Sync is pulling and then pushing to the remote. It's a function special to VS Code.

See the file helpful_git_commands.txt for code to do the above commmands.

Recommendation for Inaugral Project: if you are new to git, just work as you normally would and then commit the final product to your repo when you are done.

8. Summary

This lecture: We have discussed

1. Structuring and commenting on code
2. Debugging (try-except, assert, warnings)
3. Writing and running Python in VSCode
4. Git (version control)

Note deadline for hand-in of inaugural project is 27th of March.