Using magic commands in the Jupyter notebook can greatly increase our work efficiency
In this post, I will display some most widely used or top magic commands and some of their alternatives on different computer operating systems.
1. Magic Commands
Magic commands, also shortly called magics, are some convenient commands built into the IPython kernel, which can help easily perform particular tasks, such as interacting Python’s capabilities with the operating system, other programming languages, or a kernel. There are many magic commands in the Jupyter notebook, but in general, we can divide them into two types:
- Line Magics: the commands applied to only one line in the Jupyter cell
- Cell Magics: the commands applied to the whole cell of the notebook
In the following sections, we see the most widely used ones in these two categories. Besides, there are also some alternative commands that work similarly with some magics.
2. Line Magics
As its name suggested, a line magic is a command working for one line, which is prefixed with a single % character. We can classify them into different categories.
2.1 Information Magics
Information magics give us all information about the magic commands available in the Jupyter Notebook. There are three information magics, namely %lsmagic, %magic and %quickref.
(1) %lsmagic
There is no space between % and lsmagic, and this magic gives us a list of the whole available magics in the Jupyter Notebook.
%lsmagicIt generates the following magic information list:
Available line magics:
%alias %alias_magic %autoawait %autocall %automagic %autosave %bookmark %cd %clear %cls %colors %conda %config %connect_info %copy %ddir %debug %dhist %dirs %doctest_mode %echo %ed %edit %env %gui %hist %history %killbgscripts %ldir %less %load %load_ext %loadpy %logoff %logon %logstart %logstate %logstop %ls %lsmagic %macro %magic %matplotlib %mkdir %more %notebook %page %pastebin %pdb %pdef %pdoc %pfile %pinfo %pinfo2 %pip %popd %pprint %precision %prun %psearch %psource %pushd %pwd %pycat %pylab %qtconsole %quickref %recall %rehashx %reload_ext %ren %rep %rerun %reset %reset_selective %rmdir %run %save %sc %set_env %store %sx %system %tb %time %timeit %unalias %unload_ext %who %who_ls %whos %xdel %xmode
Available cell magics:
%%! %%HTML %%SVG %%bash %%capture %%cmd %%debug %%file %%html %%javascript %%js %%latex %%markdown %%perl %%prun %%pypy %%python %%python2 %%python3 %%ruby %%script %%sh %%svg %%sx %%system %%time %%timeit %%writefile
Automagic is ON, % prefix IS NOT needed for line magics.As the end of the magic list indicates above, % prefix of the line magics can be omitted. For example, you can type the follow command and run it:
lsmagicI suggest keeping % prefix to make it clear thatis a magic command.
(2) %magic
It prints information about the magics system in a pager on the bottom of the Jupyter Notebook. Run the following command:
%magicIt generates something like the following screenshot.
(3) %quickref
It gives a quick reference card in a pager with an overview of each available magic command in the Jupyter Notebook.
%quickrefIt displays something like the following screenshot.
2.2 Shortcut Magics
Some magics in the Jupyter Notebook play roles like shortcuts, such as %alias_magic, %autocall, %alias and %automagic, thus I call them Shortcut Magics. Let’s review them one by one as the follows.
(1) %alias_magic
As its name suggests, %alias_magic is used to create a customized magic, which is an alias for an existing magic (a line magic or a cell magic). You can create one by using:
%alias_magic customizedName ExstingName
For example, we create a magic %lm as the alias of %lsmagic, just run the following code.
%alias_magic %lm %lsmagicCreated `%lm as an alias for %lsmagic.
The above running result indicates that the alias magic has been created successfully. Later on, you can use %lm instead of %lsmagic. Let’s run it.
%lm
From the above result, you may also notice that %ml has been added into the magic list.
(2) %unalias
Opposite to the %alias and %alias_magic magics, %unalias is used to remove an alias. For example, run the following command to remove the one we just created.
%unalias %lmAfter running the command, it displays the following information.
%lm is not an alias
(3) %autocall
It makes you to call a function excluding the parentheses. For example, let’s run the following simple function.
def hello(firstname,lastname):
print(f'Hello, {firstname} {lastname}!')Then we run the magic.
%autocallAfter running the magic, you can call the function without parentheses as follows:
hello 'Jack', 'Smith'If you want to close this magic, just run it again.
(4) %automagic
This magic enables to call the line magic functions without including the % sign. The default settings are True, and this is why the end of magic list indicates “Automagic is ON, % prefix IS NOT needed for line magics.” When you run %automagic again, the automagic is off.
2.3 Directory Operation Magics and Alternatives
In the Jupyter Notebook, there are magic commands used to check your current working directory, display the contents in the working directory (CWD), create a new working directory, as well as change working directory.
(1) %pwd
As its name shows, %pwd is used to print working directory.
%pwdFor example, the output on Windows looks something like :
H:\\My Drive\\mybooks\\practical_jupyter_notebook
!pwd is an alternative command on Linux operating system. The different between them on Linux are:
%: provided by the IPython kernel and allows you to run “magic commands”
!: provided by Jupyter, which allows shell commands to be run within cells.
But `!pwd' is not recognized as an internal or external command, operable program or batch file on Windows.
(2) %ls
As its name suggests, %ls is used to list all the files in the current working directory of the Jupyter notebook.
%lsFor example, after running the above command, it displays all the files in my current working directory. Because I am using Widows, it shows something as follows:
Beside this magic, we can also use !ls on Linux, but it is not recognized as an internal or external command, operable program or batch file on Windows.
If you just want to display one type of file, for example, say .py file, you can do as follows:
%ls *.pyYou can use ls *.py alternative on Linux, but it does not work on Windows.
(3) %mkdir
As its name suggests, the %mkdir magic is used to make a new directory. For example, we create a data folder in the currently working directory (CWD) by the following code.
%mkdir dataIf the file name already existed in your directory, it will print an output something like mkdir: cannot create directory ‘foldername’: File exists.
(4) %rmdir
Opposite to mkdir magic, %rmdir folderName is used to remove a directory. For example, first, we make a testfolder in the currently working directory (CWD).
%mkdir testfolder
Then, we can run the following command to remove it.
%rmdir testfolder
Be careful to use %rmdir if there are important files in the folder, because the folder is deleted from your working directory without any notice.
(5) %cd
As its name suggests, the %cd magic is used to change your working diretory.
On Linux:
We use slash “/” on Linux operating system, for example,
%cd /home/sigmund/Documents
On Windows:
Whereas, we use backslash “\” or double one “\\”. For example,
%cd D:\sigmund\Documents
Or:
%cd D:\\sigmund\\Documents
Besides, the Shell command !cd works both on Linux and Windows.
(6) %bookmark
The %bookmark magic is used to create an alias to the current directory. Let’s run the following codes, for example.
First, let’s change the working directory to the path you like. In this example, I change it to the test folder:
%cd D:\workspace\data\testThen, we run the %bookmark magic.
%bookmark test D:\workspace\data\testLater on, you can change your working directory to ‘test’ folder by just typing %cd test. We can run %bookmark? to see more options. This magic is very helpful when you often need to switch directories from one to others.
2.4 System Operation Magics
The Jupyter Notebook provides some magics for system operation, such as package installation, check package versions and other information.
%pip magic
This magic command is used to install Python libraries and packages in the Jupyter Notebook directly rather than in terminal. The magic is used in the following way:
%pip install <packageName>We can check package version and information by using:
%pip show <packageName>For example, we install Pandas package and check its information in a Jupyter notebook by
%pip install pandas %pip show pandas
We can use the following Shell commands on both Windows and Linux,
!pip install <packageName>
!pip show <packageName>Besides, it also works without % and !.
pip install <packageName>
pip show <packageName>2.5 Magics on Python Files
For this section, there are 2 detailed posts: Different Methods to Create Python Files in Jupyter Notebook and Different Methods to Edit and Run Python Files in Jupyter Notebook. So I will not discuss these topics in detail in this post.
(1) %run
The %run is used to run python (.py) files and external Jupyter notebook (.ipynb) files in notebook cell
(2) %load and %loadpy
These two magics are used to load a .py file into current code cell of the Jupyter Notebook, and then you can edit and run it as an internal code.
2.6 Variable Magics
(1) %who
This magic is used to list all variables of global scope, and it will list all variables existing in the global scope if there is no any arguments specified. Let’s see the following example.
First, we define some variables.
a = 2 b = "name" c = 0.5
And then run the magic.
%who
We will see the following result.
a b c
When we run the following command,
%who str
it will print only the string variables
b
If we type the following
%who float
we will get only the float
c
Similarly, if you type %who int, you will obtain the result of the integer of a.
(2) %whos
Similar to %who, but it gives a bit of detailed information about each variable.
%whos
We get something like the following:
Variable Type Data/Info ----------------------------- a int 2 b str name c float 0.5
(3) %who_ls
This magic returns a sorted list of all interactive variables.
%who_ls
It will print the following result:
['a', 'b', 'c']
2.7 Timing Line Magics
There are 2 IPython Magic commands, which are used to calculate the time used to run a line of code.
(1) %time magic
The %time magic measures the execution time of a single run of the code. It gives you the information about the wall time of execution. Let’s see a simple example as follows.
%time results=[x**3 for x in range(100000)]
The result on my computer is :
Wall time: 33 ms
(2) %timeit
The %timeit magic uses the Python timeit module, which runs a statement of 7 runs, 10 loops each (by default) and then provides the best of each iteration in each round and gives time measurement with the range of standard deviation.
%timeit results=[x**3 for x in range(100000)]
The results of running on my computer are:
29.1 ms ± 620 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
2.8 History Magics
(1) %history
The %history magic displays and manipulates the history of running commands in Jupyter notebook.
%history
We can limit the numbers to show, for example, the following command shows the last five commands:
%history -l 5
(2) %dhist
It displays a history of all visited directories.
%dhist
3. Cell Magics
As its name suggested, a cell magic is a command working for a cell, which is prefixed with a double %% character.
Unlike the line magics, you cannot omit the %% prefix. The cell magics include the following ones:
%%! %%HTML %%SVG %%bash %%capture %%debug %%file %%javascript %%js %%latex %%markdown %%perl %%prun %%pypy %%python %%python2 %%python3 %%ruby %%script %%sh %%svg %%sx %%system %%time %%timeit %%writefile
3.1 Bash code magics
They work on Linux, but do not work on Windows
(1) %%!
This magic executes the whole cell in a bash and returns a list.
%%! ls
(2) %%bash
The %%bash cell magic is an extension of the ! shell prefix. It lets you run multiline bash code in the Notebook.
%%bash ls
3.2 Programming Magics
The basic structure is %%+language. We have already learned some cell magics in previous posts, such as:
%%HTMLor%%html: run HTML in the notebook cells%%writefileand%%file: create python.pyfiles, and append codes to an existing.pyfile- %%html or %%HTML: Creat and run HTML codes
- %%js or%%javascript: run JavaScript in the Jupyter notebook
Besides, R also can run in the Jupyter note using %%R after installing rpy2 library. If you are interested in running R in the Jupyter notebook, please read the post Run R in Jupyter Notebook.
3.3 Timing Cell Magics
We have discussed timing line magic above. Besides, there are also timing cell magics. The cell timing cell magics are easy to use because you just put them at the beginning of a code snippet. They are especially helpful when we test the speeds of two methods and train a machine learning model. Let’s see two simple examples.
(1) %%time
Similar with %time line magic, it gives the wall time.
%%time
# define a function
def area_triangle(a,b,c):
s = .5(a + b + c)
A = (s(s-a)(s-b)(s-c))**.5
print("Area of a triangle with sides {},{},{} is {}.".format(a, b, c, A))
# call the function
area_triangle(3,4,5)The result goes as follows:
Area of a triangle with sides 3,4,5 is 6.0. Wall time: 0 ns
(2) %%timeit
%%timeit
def area_triangle(a,b,c):
s = .5*(a + b + c)
A = (s*(s-a)*(s-b)*(s-c))**.5
print("Area of a triangle with sides {},{},{} is {}.".format(a, b, c, A))
area_triangle(3,4,5)It will give all running results and then gives the time. The results looks something as follows:
. . . Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. Area of a triangle with sides 3,4,5 is 6.0. 4.41 µs ± 230 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
Thank You For Your Vote!
Sorry You have Already Voted!
