Tools for dealing with backbone

This repository serves as a collection of tools. Currently it sports the following:

• conversion from pypsa networks to backbone inputs (.xlsx)
  • This allows for leveraging their weather-, transmission- and powerplant data aggregation utilities.
• running backbone with the converted input
• visualization of backbone results

Setup

1. install Git
2. install miniconda and make sure to add it to the PATH!
3. install GAMS, add the path of the folder in which GAMS.exe is located to your PATH variable and get a license from your supervisor.
4. optional: install Visual Studio Code
   • helpful settings:
     • CTRL + SHIFT + P lets you enter any setting
     • auto save on
     • extend number of lines shown in terminal: Files > Preferences > Settings > Feature > Terminal > Integrated: Scrollback > change value
   • optional: select default shell: git bash
   • handy extensions:
     • Git Graph; Geo Data Viewer; Jupyter; Python; Snakemake Language; Pylance; gms
     • note, that for a Jupyter Notebook to run inside vscode, you need the ipykernel package installed in an environment.
5. clone the repo: git clone --recursive https://gitlab.ruhr-uni-bochum.de/ee/backbone-tools.git
6. change into it: cd backbone-tools
7. optional but recommended: install mamba: conda install mamba -n base -c conda-forge. If you don't, use conda instead of mamba in the following commands. (will be much slower)
8. activate base environment activate base and subsequently install pypsa-environment: mamba env create -n pypsa-eur -f complete_env.yaml
9. activate the environment conda activate pypsa-eur
10. copy+paste the file config.default.yaml and rename it to config.yaml
11. install an academic license from gurobi
12. done. you can now use the tool e.g. snakemake -j8 results/elec_s_20_ec_lcopt_6H.xlsx to create a network of europe consisting of 20 nodes at a 6 hour time resolution.
13. If you experience memory errors, add --resources mem_mb=5000 to the snakemake command (you can vary the number, a smaller number uses less memory but also parallizes less processes).

Further requirements

• GAMS must be added to your environment variables such that it is callable via terminal. You can verify that by simply typing gams in a terminal. (How to add the GAMS path to the PATH variable, you can see here).

Notes

We use pypsa-eur and backbone as submodules. These may need to be initialized after cloning this without the repo without the --recursive option or when switching from a branch via git submodule update --init. After that, you can install the environment via mamba env create -n pypsa-eur -f complete_env.yaml Please note that mamba can significantly improve the installation speed of complex environments.

Once the environment and GAMS are installed, run python setup.py which will automagically add GAMS and some other required packages to the environment.

Running the conversion with snakemake

With activated pypsa-eur environment you can run rules of the provided snakefile. It allows to batch the entire process of running pypsa, converting the results to .xlsx, running backbone and finally plotting results if desired. However, you can also run the process to create a desired output by simply providing the desired file name e.g:

• snakemake -j8 pypsa-eur/networks/elec_s_20_ec_lcopt_6H.nc creates the pypsa network,
• snakemake -j8 results/elec_s_20_ec_lcopt_6H.xlsx creates the backbone input
• snakemake -j8 results/elec_s_20_ec_lcopt_6H.gdx creates the result, computed by backbone

Running only the network conversion

• You need a pypsa-network for this to work.
• make sure that your pypsa_model_path in your config file points to this network
• run pypsa_to_backbone.py or:
• run python pypsa_to_backbone.py INPUT_FILE OUTPUT_FILE, where INPUT_FILE is the pypsa-network and OUTPUT_FILE the path to the desired output

Analysing Backbone results

For analysing the results of backbone, several utility functions can be used. It is generally recommended to work with jupyter notebooks, on the top level of this repository. For instance, after cloning and opening this repository you could create a file sandbox.ipynb, or (if using vs-code) create a new notebook with CTRL + SHIFT + P.

Loading a result

from scripts.results import BackboneResult
import plotly.express as px

# this file must be present in your results directory for this to work
result = BackboneResult("results/elec_s_5_ec_lcopt_Co2L-24H_result.gdx")

# any result symbol can be accessed like this
energy_generation = result.r_gen()

# turn the timestep strings to integers by dropping the first letter
energy_generation["t"] =  energy_generation["t"].apply(lambda x: int(x[1:]))
energy_generation.head()

This should yield an output like:

If you have plotly installed, you can easily visualize data that is in long data format.

# this plot looks better, if the temporal resolution is <12h
px.line(energy_generation, x="t",y="Val",color="unit")

Converting from long to wide data format

By default, backbone outputs are in long data format. To transform that into wide data format, the following method can be used:

# limit data to electricity
electricity_generation = energy_generation.query("grid == 'elec'")

# drop columns
electricity_generation.drop(["f","grid","node"],axis=1,inplace=True)

# pivot to wide data format
wide_form_r_gen = electricity_generation.pivot(index="t",columns="unit",values="Val")

# this can be plotted with e.g. matplotlib
wide_form_r_gen.plot.area()

Writing results to excel

You can also use gdxxrw in GAMS to write Backbone results to an excel file. Use writeExcel.gms (in the backbone tutorial files) or generate a new .gms file. Adjust the paths and result parameter to your needs.

$call 'gdxxrw "path\to\backbone\output\results.gdx" output="path\to\backbone\output\results_r_gen.xlsx" par=r_gen rng=A1'

Note, that you can also execute the command

gdxxrw "path\to\backbone\output\results.gdx" output="path\to\backbone\output\results_r_gen.xlsx" par=r_gen rng=A1

in a terminal.

Questions?

Please consult our wiki pages, if you run into any troubles. You can also check out the wiki of backbone.