Data Engineering/Systems/Jupyter/Tips
Spark config
Application Master
Spark can be run either in local mode, where all execution happens on the machine the notebook is started from, or in Yarn mode which is fully distributed.
Local mode is especially convenient for initial notebook development. Local mode is typically more
responsive than Yarn mode, and allows prototyping code against small samples of the data before
scaling up to full month or multi-month datasets. Dependencies are additionally easily handled
in local mode, when running local nothing special has to be done.
Dependencies can be installed with !pip install
and will be available in all
executor processes. In local mode the number of java threads can be specified using brackets,
such as local[12]
. It is recommended to provide more memory than the defaults when
running in local mode, otherwise a dozen java executor threads might fight over a couple GB of memory.
Note: Using spark in local mode implies some knowledge of the resources of the local machine it runs on (number of CPU cores and amount of memory). Please don't use the full capacity of local machines! Some other users might be working with it as well :)
(
builder
.master('local[12]')
.config('spark.driver.memory', '8g')
)
Local mode has it's limits. When more compute is necessary set the master to yarn and specify
some limits to the amount of compute to use. Spark will execute a task per core, so 64 executors
with 4 cores will give 256 parallel tasks with 64 * 8G = 512G of total memory to work on the
notebook (this represent ~15% of the whole cluster). Finally some more memory is needed for
the spark-driver: 2g instead of 1g as default. The driver being responsible to collect
computed data onto the notebook (if needed) as well as managing the whole tasks the
application deals with, some memory shall prevent it to crash too quickly.
(
builder
.master('yarn')
.config('spark.dynamicAllocation.maxExecutors', 64)
.config('spark.executor.memory', '4g')
.config('spark.executor.cores', 4)
.config('spark.driver.memory', 2g)
)
Other working examples
An example notebook has been put together demonstrating loading the ELMo package from tensorflow hub and running it across many executors in the yarn cluster.
Sharing Notebooks
There is currently no direction functionality to view (Phab:T156980) or share
(Phab:T156934) other users' notebooks in real-time, but it is possible to copy notebooks
and files directly on the server by clicking 'New' -> 'Terminal' (in the root folder in the
browser window) and using the cp
command.
GitHub
It's also possible to track your notebooks in Git and push them to GitHub, which will display them fully rendered on its website. If you want to do this, you should connect to GitHub using HTTPS. SRE recommends not using SSH because of the risk that other users could access your SSH keys (which, if combined with a production SSH key reused for GitHub, could result in a serious security breach).
With HTTPS, by default you'll have to type in your GitHub username and password every time you
push. You can avoid this by adding the following
(from this Superuser answer) to ~/.gitconfig
:
[url "https://YOURUSERNAME@github.com"] insteadOf = https://github.com
[credential] helper = cache --timeout=28800
This will automatically apply your GitHub user name to any HTTPS access, and then cache the password you enter for 8 hours (28 800 seconds) such that you need to enter it only once in a working day.
You can also set up a personal access token to use for authentication over HTTPS. This will allow you to not enter your GitHub password every time. These tokens can also be limited in what access they have, e.g. they can be set to only be able to modify repositories.
HTML files
You can also export your notebook as an HTML file, using
File
> Download as...
in the JupyterLab interface or
the jupyter nbconvert --to html
command.
If you want to make the HTML file public on the web, you can use the web publication workflow.
HTTP requests
Allow HTTP requests, to for example enable your notebook to clone a repo, by adding export code to your .bash_profile in the Terminal Notebook.
Sending emails from within a Notebook
To send out an email from a Python notebook (e.g. as a notification that a long-running query or calculation has completed), one can use the following code
In[1]:
notebookservername = !hostname
notebookserverdomain = notebookservername[0]+'.eqiad.wmnet'
username = !whoami
import smtplib
def send_email(subject, body, to_email = username[0]+'@wikimedia.org', from_email = username[0]+'@'+notebookserverdomain):
smtp = smtplib.SMTP("localhost")
message = """From: <{}>
To: <{}>
Subject: {}
{}
""".format(from_email, to_email, subject, body)
smtp.sendmail(from_email, [to_email], message)
# example uses:
# send_email('Jupyter notebook ready (n/t)', <nowiki>''</nowiki>)
# send_email('Jupyter email test', 'test body', 'yourname@wikimedia.org', 'yourname@wikimedia.org')
(Invoking the standard mail client via the shell, i.e. !mailx
or !heirloom-mailx
,
fails for some reason, see phab:T168103.)
Using R in Python through rpy2
rpy2 enables access to R in Python, and is particularly useful in Jupyter notebooks in that you can for instance use ggplot2 for your visualizations. As of March 2020, installing rpy2 on the Jupyter servers is not necessarily straightforward as the latest version of it is incompatible. Using a version lower than 3.1 appears to work, e.g.
pip install --upgrade "rpy2 < 3.1" pip install tzlocal
The second call is needed because this version of rpy2 fails to install the required "tzlocal" package. Once installed, you should be able to load the extension with the following code in a notebook:
%load_ext rpy2.ipython
And once that is done, use the %R
and %%R
magic words to execute R code in the notebook.