Talk:Data Engineering/Systems/Conda

Libmamba solver

I've been trying to get libmamba solver to work with my conda env, and I don't think the instructions provided in this doc are correct. On the libmamba page, it says "The instructions in this page assume you are using conda-libmamba-solver 22.12 or above." Our conda distribution for conda-analytics is currently 4.13.0, and running conda update -n base conda to update conda to a newer version doesn't work because of denied permissions. This (for the moment, at least) renders the libmamba documentation incorrect, because libmamba can't be used with our internal systems. Htriedman (talk) 19:37, 12 July 2023 (UTC)

running conda config --set solver libmamba without a conda upgrade first yields the following error: CondaValueError: Key 'solver' is not a known primitive parameter. Htriedman (talk) 19:40, 12 July 2023 (UTC)

Using Jupyter to migrate from anaconda-wmf to conda-analytics

From Jupyter Console:

1. Select File -> Hub Control Panel This will open a new tab with ‘Stop My Server’ and ‘Start My Server’ buttons.
2. Select ‘Stop My Server’ and then, when ready, ‘Start My Server’
3. From the ‘Select a job profile:’ pulldown menu select ‘Create and use new cloned conda environment…’ The ‘Spawning server…’ progress bar will run, once completed you should get a Server Ready message, tab will open a new console window. If the initial cloning action fails you may need to logout and re-try again.
4. To test that you are indeed operating in conda-analytics open a Terminal and run command `conda list | grep spark`

If clone was successful you will see the following version numbers:

findspark                 2.0.1                  

pyspark                   3.1.2

When you next login to use Jupyter you should be operating in the new conda-analytics env.

Remember to create the same new conda-analytics envs in the other stat servers you have access to. Shay Nowick (talk) 23:50, 22 February 2023 (UTC)

@Shay Nowick thank you for writing these great instructions! I think you should add them to the page itself rather than just to the talk page 😊 Neil P. Quinn-WMF (talk) 09:16, 23 February 2023 (UTC)

Done, thanks! Shay Nowick (talk) 20:31, 23 February 2023 (UTC)

+1 to what @Neil P. Quinn-WMF said Bearloga (talk) 15:20, 23 February 2023 (UTC)