Welcome to eTraGo’s documentation!

Warning

Note, eTraGo and its documentation is in continuous development.

About eTraGo

eTraGo stands for electric Transmission Grid optimization.

The python package eTraGo provides optimization strategies of flexibility options for transmission grids based on PyPSA. A peculiarity in this context is that the German transmission grid is described by the 380, 220 and 110 kV voltage levels. Conventionally the 110kV grid is part of the distribution grid. The integration of the transmission and ‘upper’ distribution grid is part of eTraGo.

The focus of optimization are flexibility options with a special focus on energy storage and grid expansion measures.

The open_eGo project

This software project is part of the research project open_eGo.

The OpenEnergy Platform

Within this project we developed the OpenEnergy Platform which this software is using in order to get and store the in- and output data. Before you start to calculate a registration on the platform is needed. For more information see openenergy-platform and login.

The OpenEnergy platform mainly addresses students, researchers and scientists in the field of energy modelling and analytics as well as interested persons in those fields. The platform provides great tools to make your energy system modelling process transparent. All data of the open_eGo project are stored at this platform. Learn more about the database access.

Model overview

eDisGo

The python package eDisGo provides a toolbox for analysis and optimization of distribution grids. It is closely related to the python project Ding0 as this project is currently the single data source for eDisGo providing synthetic grid data for whole Germany. Learn more here.

eGo

The python package eGo is a toolbox and application which connects the tool eTraGo (optimization of flexibility options at transmission grid level) and eDisGo (optimization of distribution grids). All those python packages are part of the research project open_eGo. Learn more here.

Dataprocessing

For the open_eGo project several python packages are developed which are feeded by the input data of the data processing. The Dataprocessing is writen in SQL and Python. Learn more here.

ego.io

The ego.io serves as a SQLAlchemy Interface to the OpenEnergy database (oedb). The oedb table ORM objects are defined here and small helpers for io tasks are contained. Learn more here.

Dingo

The DIstribution Network GeneratOr (Ding0) is a tool to generate synthetic medium and low voltage power distribution grids based on open (or at least accessible) data. Learn more here.

LICENSE

© Copyright 2015-2018 Flensburg University of Applied Sciences, Europa-Universität Flensburg, Centre for Sustainable Energy Systems and DLR-Institute for Networked Energy Systems

This program is free software: you can redistribute it and/or modify it under the terms of the GNU Affero General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more details.

You should have received a copy of the GNU General Public License along with this program. If not, see www.gnu.org/licenses.

Installation

eTraGo is designed as a Python package therefore it is mandatory to have Python 3 installed. If you have a working Python3 environment, use pypi to install the latest eTraGo version. We highly recommend you to use a virtual environment. Use following pip command in order to install eTraGo.

$ pip3 install eTraGo

Using a virtual environment

Before installing eTraGo, you create a virtual environment (where you like it) and activate it:

$ virtualenv venv --clear -p python3.7
$ source venv/bin/activate
$ cd venv

Inside your activated virtual environment you can install eTraGo with the pip command, as previously explained.

Linux and Ubuntu

The Package eTraGo is tested with Ubuntu 16.04, 18.04 and 20.04 inside the virtual environments of virtualenv. The installation is shown above.

Windows or Mac OSX users

For Windows and/or Mac OSX user we highly recommend to install and use Anaconda for you Python3 installation. First install anaconda including python 3.5 or higher version from https://www.anaconda.com/download/ and open an anaconda prompt as administrator and run:

$ conda config --add channels conda-forge
$ conda install shapely
$ pip install eTraGo

The full Documentation can be found on this page . We use Anaconda with an own environment in order to reduze problems with Packages and different versions on our system. Learn more about Anacona environments.

Setup database connection

The package ego.io will be installed automatically when eTraGo is installed. The egoio gives you python SQL-Alchemy representations of the OpenEnergy-Database(oedb) and access to it by using the oedialect, which is a SQL-Alchemy binding Python package for the REST-API used by the OpenEnergy Platform (OEP).

In order to connect eTraGo via the oedialect with the oedb you have to create an account at openenergy-platform.org/login. You can name the ‘db’ argument of the ‘args’ of the etrago.appl.etrago() as you wish. Once the etrago.appl.etrago() is executed you will be asked to enter how you want to connect to which database. If you want to use the oedialect enter the following connection parameter. For <username> and <token> you have to take your credentials which you obtained by registering at openenergy-platform.org/login.

Your API access / login data will be saved in the folder .egoio in the file config.ini. Consequently, in the config.ini you can also change your connection parameters or add new ones. In the following you can see how the config.ini looks like when you use the oedialect, a local postgresql database or the old psycopg2 developer connection.

Once you have created a connection (which is saved in the config.ini) you do not have to enter the connection parameter again. The software will take the connection parameter which corresponds to the entry at the ‘db’ argument.

oedialect connection

[oedb]
dialect  = oedialect
username = <username>
database = oedb
host     = openenergy-platform.org
port     = 80
password = <token>

Local database connection

[local]
username = YourOEDBUserName
database = YourLocalDatabaseName
host = localhost or 127.0.0.1
port = 5433
pw = YourLocalPassword

Old developer connection

[oedb]
username = YourOEDBUserName
database = oedb
host = oe2.iws.cs.ovgu.de
port = 5432
pw = YourOEDBPassword

How to use eTraGo?

After you installed eTraGo you would typically start optimization runs by executing the ‘appl.py’ which is situated in ./eTrago/etrago/ (e.g by python3 appl.py).

eTraGo doesn’t have a graphical user interface, the ‘appl.py’ is used as a simple user interface which can be edited with the preferred python-editor. Here parameters, calculation methods and scenario settings are set in a python dictionary called ‘args’. To run the desired calculation, it is crucial to understand these parameters. In addition, some of them contradict the usage of others. You find the documentation of all defined parameters from the ‘args’ here: etrago.appl.run_etrago().

Alternatively, the ‘args’ dictionary can be edited in a json-file. Then the path to the json-file has to be set in the initilization of the Etrago-object (etrago.tools.network.Etrago). Once a path is given the ‘args’ dictionary within the ‘appl.py’ is ignored and replaced by the ‘args’ of the json-file.

The appl.py contains the etrago.appl.run_etrago() function which uses the defined ‘args’ dictionary to start the desired calculation.

To improve the performance of the optimization of the selected solver, you might want to use solver options (part of ‘args’). For gurobi the most used ones are described here.

For more specific or extensive changes you are highly invited to write code and add new functionalities.

Once the calculation has finished the PyPSA network of the Etrago-object will contain all results. Som main results (e.g. anuual system costs) are calculated by etrago.calc_results() and can be accesed via ‘etrago.results’. You can use several plotting functions from the etrago.tools.plot() in order to visualize the results. For example the etrago.tools.plot.plot_grid() can be used to plot relative line loading in % or the optimized expansion of all AC lines and DC links of the network.

To save the results you can write them to csv files. These functionalites can be specified also in the ‘args’ dictionary.

Examples and tutorial notebooks

Theoretical Background

Definitions and Units

eTraGo executes the Open Source software PyPSA to perform power flow simulations and uses its definitions and units.

Assumptions on Data

eTraGo fetches its necessary input data from the OpenEnergy Platform including load, generation, grid and scenario-related data. More details can be found in the Data-Processing.

As overview, the Open Source grid structure is developed by processing data from OpenStreetMap (OSM) to obtain geo-referenced locations of substations and links equal or above the 110 kV voltage level. OSM also provides information about residential, retail, industrial and agricultural areas which is used with standardized profiles to obtain load data. Generation data of solar and wind rely on weather data from [coastdat-2]. Both, load and generation data, match the annual amount for the year 2011. eTraGo enables the investigation of three scenarios - Status Quo, NEP 2035 and eGo100. Status Quo corresponds to the actual grid, NEP2035 follows assumptions for the year 2035 by [NEP2015] and eGo100 assumes to operate the future energy system completely by renewables [ehighway2050].

Methods

PyPSA

The power flow simulations are performed by the Open Source tool PyPSA with a linear approximation for the optimization of power flows in general. Expecting that eTraGo fulfills the assumptions to perfom a LOPF (small voltage angle differences, branch resistances negligible to their reactances, voltage magnitudes can be kept at nominal values) since it focuses on the extra-high and high voltage levels. As objective value of the optimization, the overall system costs are considered.

Clustering approaches

EHV-Clustering

This method maps an input network to an output network with the nodes of the extra-high voltage level. All nodes with a voltage level below the extra-high voltage level are mapped to their nearest neighboring node in the extra-high voltage level with the dijkstra algorithm (110 kV —> 220,380 kV).

K-Means Clustering

This method maps an input network to a new output network with an adjustable number of nodes and new coordinates. The algorithm sets these coordinates randomly and minimizes a certain parameter like for example the distances between old coordinates and their nearest neighbor in the set of new coordinates. The method was implemented by Hoersch et al. within PyPSA.

Snapshot skipping

This method simplifies the simulation temporally by considering every n-th snapshot of a given time series. The regarded snapshots are weighted by the number of neglected snapshots to ensure a comparable calculation of costs. This method assumes the chosen snapshots to be represenative for the next n-th snapshots in the time series.

Snapshot-Clustering

This method aggregate given time series for various time intervals like i.e. days using the tsam package. Contrary to snapshot skipping, this approach averages a certain period of snapshots instead of choosing a representative snapshot.

Storage expansion

To evaluate the amount of storage units in future energy systems, the possible installation of new storage units at every node in the network is allowed. The size and operation of these storages are part of the optimization problem.

Two types of storage technologies are considered - batteries and hydrogen in underground caverns. Li-Ion battery storages as representatives for short-term (several hours) technologies, which can be installed at every node. Underground hydrogen storages represent long-term or seasonal (weeks) technologies and can be build at every node with appropriate salt formations in the underground. The storage parameters for both types are reached by [Acatech2015], the information about salt formations are given by [BGR].

Grid expansion

The grid expansion is realized by extending the capacities of existing lines and substations. These capacities are regarded as part of the optimization problem, whereby the possible extension is unlimited. With respect to the different voltage levels and lengths MVA-specific costs are considered in the linear optimization of the power flow. Besides, several planned grid expansion scenarios from the German grid development plan can be considered as possible additional power lines by using the ‘scn_extension’ argument.

Miscellaneous Features

Several features were developed to enhance the functionality of eTraGo. As appropriate computer setting, the ‘solver_options’ and a ‘generator_noise’ are possible arguments. The latter adds a reproducible small random noise to the marginal costs of each generator in order to prevent an optima plateau. The specific solver options depend on the applied solver like for example Gurobi, CPLEX or GLPK. Considering reproducibility, the ‘load_cluster’ argument enables to load a former calculated clustered network. Besides, ‘line_grouping’ provides a grouping of lines which connect the same buses. The ‘branch_capacity_factor’ adds a factor to adapt all line capacities in order to consider (n-1) security. Because the average number of HV systems is much smaller than the one of eHV lines, you can choose factors for ‘HV’ and ‘eHV’. The ‘load_shedding’ argument is used for debugging complex grids in order to avoid infeasibilities. It introduces a very expensive generator at each bus to meet the demand. When optimizing storage units and grid expansion without limiting constraints, the need for load shedding should not be existent. The ‘minimize_loading’ argument forces to minimize the loading of the lines next to the costs. ‘Parallelization’ provides the opportunity to devide the optimization problem into a given number of sub-problems. For a group of snapshots the problem will be solved separately. This functionality can only be used for problems which do not have dependencies from one snapshot to another. Therefore this option can not be used with the optimization of storage units due to their state of charge.

References

[NEP2015]

Übertragungsnetzbetreiber Deutschland. (2015).: Netzentwicklungsplan Strom 2025, Version 2015, 1. Entwurf, 2015. (https://www.netzentwicklungsplan.de/sites/default/files/paragraphs-files/NEP_2025_1_Entwurf_Teil1_0_0.pdf)

[coastdat-2]

coastDat-2 (2017).: Hindcast model http://www.coastdat.de/data/index.php.en

[ehighway2050]

e-Highway2050. (2015).: e-HIGHWAY 2050 Modular Development Plan of the Pan-European Transmission System 2050 - database per country. Retrieved from (http://www.e-highway2050.eu/fileadmin/documents/Results/e-Highway_database_per_country-08022016.xlsx)

[Acatech2015]

‘Flexibilitätskonzepte für die Stromversorgung 2050 www.acatech.de/fileadmin/user_upload/Baumstruktur_nach_Website/Acatech’

[BGR]

‘Salzstruktur in Norddeutschland <>’_. 2015.: Data provided by the Federal Institute for Geosciences and Natural Resources (Bundesanstalt für Geowissenschaften und Rohstoffe, BGR)

Developer notes

Installation for Developers

Note

Installation is primarly tested on (Ubuntu like) linux OS.

1. If you like, create a virtual environment (where you like it) and activate it (if you do not use venv start with 2.):

$ virtualenv --clear -p python3.7  etrago``
$ cd etrago/
$ source bin/activate

2. Clone the source code from github:

$ git clone https://github.com/openego/eTraGo

You can checkout to the dev branch and create new feature branches. For the correct work-flow, please mind the Dreissen Branching Model

3. Use the pip -e to install eTraGo directly from the cloned repository:

$ pip3 install -e /path/to/eTraGo/

What’s New

These are new features and improvements of note in each release.

Release 0.8.0 (April 8, 2021)

eTraGo has now a more object-oriented programming design.

Added features

• eTraGo uses PyPSA version 0.17.1 directly, the fork is not needed anymore. The updated pypsa version includes various features, e.g. running a lopf without using pyomo which is faster and needs less memory.
• (n-1)-security factors are set as line/transformer parameters s_max_pu instead of adding the additional argument s_nom_original
• There is now one central plotting function for all grid topology plots which also allows to combine different results (e.g. plot storage expansion and line expansion at once)
• eTraGo is now compatible to Python3.7
• A bug in setting the line_length_factor in kmeans clustering is fixed.

Release 0.7.2 (Juni 15, 2020)

A minor release adding the following features.

Added features

• for single use of eTraGo (not as a sub-module of eGo), we recommend to use the newest minor data release ‘gridversion’: ‘v0.4.6’. This data release includes some minor bug fixes but it is not consistent with the data on the MV and LV levels. Hence, the modelling results are only adequate for the HV and EHV level applying solely the tool eTraGo.
• snapshot clustering includes now an approach to model seasonal storage as in Kotzur et al, 2018 ( https://www.sciencedirect.com/science/article/pii/S0306261918300242 ). Moreover the method may include extreme periods using an option of the tsam package.
• osm maps can now be used for background plotting
• the iterate_lopf function enables to adequately model the reactances when expanding the grid
• important bug fix for the adjustment of reactances when harmonizing the voltage level when applying the k-means network clustering
• multiple extra_functionalities can be called easily called now at once
• various minor changes such as specifying installation requires for flawless performance

Release 0.7.1 (October 25, 2018)

A minor release adding new options for additional constraints, modelling assumptions and plotting.

Added features

• Two extra functionalities were introduced in order to apply constraints concerning a minimal share of renewable energy and a global upper bound for grid expansion. You can activate these functions in the ‘args’ of the etrago() function.
• The branch_capacity_factor can now be defined separately for the high and extra high voltage level in order to address the (n-1) criteria more accurately.
• There are some more plotting functions e.g. plotting the state-of-charge and dispatch of storage units.
• Storage capacities in foreign countries can easily be be optimized.
• By default the maximum expansion of each line and transformer is set to four times its original capacity. Being an argument of the extendable() function it can be easily adjusted.
• k-means clustered results can now also be exported to the oedb.

Release 0.7.0 (September 6, 2018)

eTraGo is able to produce feasible non-linear power flows based on optimization results and allows the disaggregation of clustered results to original spatial complexities.

Added features

• The pf_post_lopf function was improved. Due to changes in the data set now the non-linear power flow (pf) creates feasible solutions. If network optimization is turned on, a second lopf which regards the updated reactances and optimizes only dispatch is performed before the pf is executed.
• The disaggregation method was included. When using a network clustering method to reduce the spatial complexity of the given network, a disaggregation method can be used afterwards to distribute the nodal results (generation and storage timeseries) to the original complexity. The method ‘disaggregation’: ‘uniform’ can be used as an interface functionality for distribution grid planning tools like eDisGo.
• For the network expansion it is now additionally possible to only optimize the German power lines or only the crossborder lines. Moreover one can choose to optimize only a predefined set of power lines which are identified by a worst-case analysis beforehand.
• Intertemporal constraints can be applied to certain power plants. For different technologies certain parameters i.e. ‘start_up_cost’, ‘start_up_fuel’, ‘min_up_time’ and ‘min_down_time’ are defined in the ramp_limits function.
• Crossborder lines can now easily be modelled as ‘DC’ links. Moreover the capacities of these lines can be adjusted with respect to a ACER report on thermal as well as net transfer capacities.
• Thanks to @jankaeh manually the grid topology within the cities Stuttgart, Munich and Hannover was improved. Perspectively this function should be obsolete when openstreetmap and/or osmTGmod get better data coverage.
• As an alternative to the normal editing of the calcualtion settings (args) within the appl.py it is now possible to load an args.json file.

Release 0.6.1 (Juli 18, 2018)

eTraGo works with pypi and is suitable for eGo 0.2.0

Added features

• An installation issue when installing from pypi was fixed.
• The random noise function was improved. Now you set a (reproducible) random seed.
• snapshot.weightings are used within the plotting functions
• bug fix for k-means clustering with respect to the aggregation of p_max_pu values of variable generators. They are weighted by their p_nom now.

Release 0.6 (June 27, 2018)

eTraGo now enables combined grid and storage expansion, snapshot clustering and the consideration of exogenous grid expansion.

Added features

• A part from optimizing the investment of storages it is now also possible to optimize grid expansion investments. In this context we added an argument ‘extendable’ which expects an array of the different component types you want to optimize. This argument corresponds to functions in the new extendable.py file. It is possible to choose from expansion strategies which are defined within that sub-package. Capital costs for new grid components can be defined and are annualized by means of interest rate, component lifetime and operation period.
• The k-means network clustering (‘network_clustering_kmeans’) has been refactored. It is now possible to reproduce busmaps by csv-importing already defined busmaps. Consequently it is possible to write busmaps. Here the argument ‘load_cluster’ was introduced. Moreover it is possible read and write bus_weightings. This helps to e.g. run a future scenario but using a bus weighting of the status quo. Moreover, the remove_stubs function from PyPSA is now easily usable in eTraGo.
• The snapshot_clustering can now be used in order to reduce the temporal complexity. Typical days are taken in order to represent the entire year. Here the package tsam (developed by Leander Kotzur) is used. Standardly a hierarchical clustering method is used which is e.g. described by Nahmacher et al. ( see: https://www.sciencedirect.com/science/article/pii/S0360544216308556 ).
• Scenario variations from the oedb can be introduced. The argument ‘scn_extension’ will activate an extension scenario which adds components such as lines or generator to the base scenario. The ‘scn_decommissioning’ argument states whether you want to remove existing components. Right now, in the oedb two scenarios are accessible which represent the grid expansion (and the corresponding removal of existing equipment) planned by the German network development plan.
• Our data model at the oedb represents Germany and the electrical neighbors. If you consider planned grid expansion to Norway and Belgium you would most probably want to include also these countries as electrical neighbors including their aggregated generation and demand characteristics. The argument ‘add_Belgium_Norway’ therefore was introduced. Once activated it will add the countries to the model.
• DC links are now also modelled as PyPSA DC links. Consequently Sweden is now connected by a DC link.

Other changes

• The plotting sub-package was amplified and enhanced by new plotting functions and improvements of existing ones (e. g. introduced a legend for the plotting function storage_expansion())
• The code complies now mostly with the pep8 standard
• Documentation was improved (but ongoing work in progress) considering doc strings for functions and the rtd-documentation web site
• The io was slightly reformatted and restructured.

Release 0.5.1 (February 01, 2018)

eTraGo works with ego.io 0.3.0

Added features

• Result export to oedb functionality was improved. Now, a safe tag can be set in order to state that the result set shall be versioned and moved to the schema ‘grid’.
• the new database sessionmaker of ego.io 0.3.0 is enabled which gets rid of dependency to oemof.db.
• it is possible to skip snapshots in order to ingenuously simplify the problem.

Other news

• eTraGo will be available on PyPI
• eTraGo 0.5.1 will be used for eGo 0.0.1

Release 0.5 (December 08, 2017)

eTraGo works with PyPSA 0.11.0.

Added features

• Readthedocs documentation
• Result export to postgresql database in particular to the open energy data base (oedb)
• parallelisation function hands over SOC of storages to following problem.
• New plot for displaying voltage deviations at network buses.
• Line loading plot displays the direction of power flows.

Bug fixes

• k-means clustering got a more suitable scaling factor concerning calculations on the 110kV grid.
• K-means weighting of the buses is a bit more robust to changes in the data structure.
• the list of carriers is imported again to the pypsa network from the oedb data model.

Other changes

• update PyPSA Version from 0.8.0 to PyPSA 0.11.0

Release 0.4 (October 12, 2017)

eTraGo integrates ego.powerflow functionalities

Release 0.4 is mainly the merging of ego.powerflow into eTraGo. Additionally, some restructuring has been carried out, plotting functions have been updated and the first approach for a documentation was set up.

Other changes

• merging of ego.powerflow into eTraGo

Release 0.3 (September 8, 2017)

Release introducing k-means clustering and several additional functionalities

Added features

• k-means clustering

Release 0.2 (July 20, 2017)

This is the version 0.2 of eTraGo.

This new version shall be install via pip and a setup.py. Moreover there has been minor adjustments such as the creation of a callable etrago function.

Added features

• install via pip and a setup.py
• callable etrago function

Release 0.1 (June 30, 2017)

First release of eTraGo

etrago

etrago package

Subpackages

etrago.cluster package

Submodules

etrago.cluster.networkclustering module

etrago.cluster.snapshot module

Module contents

etrago.tools package

Submodules

etrago.tools.io module

etrago.tools.extendable module

etrago.tools.plot module

etrago.tools.snapshot_clustering module

etrago.tools.utilities module

Module contents

Submodules

etrago.appl module

Module contents

Indices and tables

• Index
• Module Index
• Search Page