Newsletter

Introduction

SeaDataNet is a major operational infrastructure for managing, indexing and providing access to ocean and marine data sets and data products, acquired by European organisations from research cruises and other observational activities in European coastal marine waters, regional seas and the global ocean. It develops, governs and promotes common standards for metadata and data formats, common vocabularies and quality flags and software tools and services for marine data management, which are widely adopted and used. SeaDataNet core partners are the National Oceanographic Data Centres (NODCs) and major marine research institutes in Europe. It has established a large European and international network, working closely together with operational oceanography, marine research, and marine environmental monitoring communities as well as with other marine data management infrastructures. SeaDataNet is also a major partner in the development and operation of the European Marine Observation and Data network (EMODnet), aimed at supporting the EU initiatives on Marine Knowledge 2020 and Blue Growth and the Marine Strategy Framework Directive (MSFD). Since the mid-1990s SeaDataNet has expanded and matured and at present it provides federated discovery and access to more than 110 data centres for physics, chemistry, geology, bathymetry, and biology. SeaDataNet is further developing its discovery, access, ingestion, publishing and visualisation services in the EU HORIZON 2020 SeaDataCloud project. This aims at upgrading and expanding the SeaDataNet architecture and services, making use of cloud services, taking into account the European Open Science Cloud (EOSC) challenge. This is done in a partnership with EUDAT, a leading European network of academic computing centres. The major objectives of the SeaDataCloud project are:

• Improve discovery and access services for users and data providers
• Optimise connecting data providers and their data centres and data streams to the infrastructure
• Improve interoperability with other European and international networks to provide users overview and access to additional data sources
• Develop a Virtual Research Environment with tools for analyzing data and generating and publishing data products.

This is the third edition of the newsletter in the framework of the SeaDataCloud project. It gives you information about the progress of a number of SeaDataCloud developments which after circa 30 months have reached a critical phase of implementation. The Newsletter also gives information on related projects, such as EMODnet developments with involvement of SeaDataNet. We hope you will enjoy this newsletter and will be triggered to visit the SeaDataNet portal for a try-out of its services and to follow its evolution. We aim to reach as many people as possible, so please forward it to anyone you know may be interested.

Progress with population of SeaDataNet directories

The SeaDataNet infrastructure comprises a network of interconnected data centres that perform marine data management at national and local levels and that together make their information and data resources discoverable and accessible in a harmonized way. The SeaDataNet directory services provide overviews of marine organisations in Europe, and their engagement in marine research projects, managing large datasets, and data acquisition by research vessels and monitoring programmes for the European seas and global oceans:

• European Directory of Marine Organisations (EDMO) (> 4.100 entries)
• European Directory of Marine Environmental Data (EDMED) (> 4.200 entries)
• European Directory of Marine Environmental Research Projects (EDMERP) (> 3.100 entries)
• European Directory of Cruise Summary Reports (CSR) (> 57.000 entries)
• European Directory of the Ocean Observing Systems (EDIOS) (> 10.000 entries)
• Common Data Index Data Discovery and Access service (CDI) (>2.3 million entries)

 
Figure: the monthly progress of each of the directories since September 2012.

Implementing the upgraded SeaDataNet CDI service

The SeaDataNet CDI Data Discovery and Access service has been built up and its network has expanded since 2000 with support of many EU projects such as Sea-Search, EuroCore, EuMarsin, EuroSeismics, SeaDataNet, BlackSeaScene, Upgrade-BlackSeaScene, Geo-Seas, MicroB3, and in the last 10 years as part of SeaDataNet 2, ODIP 1 & 2, and EMODnet projects.

The SeaDataNet Common Data Index (CDI) service gives harmonized discovery and access to a large volume of marine and ocean data sets, both from research and monitoring organisations, which increasingly are major input for developing added-value services and products that serve users from government, research and industry. Major products are for instance developed by EMODnet lots, Copernicus Marine Environmental Monitoring Service (CMEMS), and SeaDataNet itself. To illustrate: EMODnet Chemistry delivers harmonized and validated aggregated data collections for eutrophication, contaminants and marine litter for all European sea basins which serve the Marine Strategy Framework Directive (MSFD and have been taken up by Regional sea Conventions, EU DG-Environment and European Environment Agency (EEA) for supporting assessments and deriving indicators. EEA is actively encouraging more member states to adopt EMODnet Chemistry (and thus the SeaDataNet CDI service) to make their environmental monitoring data available for the MSFD process. EMODnet Bathymetry generates the market leading Digital Terrain Model (DTM) for the European seas which finds its way towards many numerical modellers for tide and wave forecasting, offshore industry such as oil & gas companies, wind farm operators, dredging companies, pipeline engineering companies, coastal protection managers, and many others. CMEMS deploys pan-European capacity for Ocean Monitoring and Forecasting. SeaDataNet and CMEMS have an MOU in place for mutual exchanges of data, adoption by CMEMS of SeaDataNet standards, and for developing joint products such as climatologies. Next to serving organised communities, the SeaDataNet CDI service also serves individual users, mostly from research sectors. This is further stimulated, inter alia through cooperation with the evolving European Open Science Cloud (EOSC).

At present the SeaDataNet CDI service has > 110 connected data centres, and provides metadata and access to more than 2.3 Million data sets, originating from more than 650 organisations in Europe, covering physical, geological, chemical, biological and geophysical data, and acquired in European waters and global oceans.

Figure: Overview of CDI entries per July 2019: >2.3 million data sets from 650+ originators and 110+ connected data centres

Figure: Number of CDI entries per July 2019 per discipline

Figure: Number of CDI entries per July 2019 per year of observation

In the ongoing HORIZON 2020 SeaDataCloud project extensive activities have been undertaken for increasing performance and to overcome experienced issues of the current SeaDataNet CDI service. This is achieved by upgrading the CDI service by integrating a cloud environment and further developing the services and protocols, both for ingestion of new and updated CDI entries by connected data providers as well as discovery and downloading of data sets by users. For this upgrading SeaDataNet technicians works together with EUDAT technicians, who are also largely involved in the development of EOSC.

Figure: Upgraded CDI service structure

Data cloud serving as data cache with replicates of unrestricted data sets and with PIDs for improving delivery to users by one data package per shopping basket and without waiting for services of data centres Replication Manager replacing the Download Manager at data centres for ingesting new metadata - data sets and mediating delivery of restricted data sets after approval of data centres Increased quality control of formats and semantics as used by data centres for ingested meta and data and their mutual coherence New user interface with MySeaDataNet functionality to improve and ease user experience

In the new architecture a separation has been made between the front end with discovery, shopping and downloading of unrestricted data sets by users, and the back end for importing new and updated CDI metadata and related data entries (including versioning) by data centres.

The front end is a modern user interface with a combination of searching by controlled criteria and downdrilling by facets, while Elastic Search is applied to give high performance free search on all CDI details. This in combination with a mapping tool based upon GeoServer and OpenLayers gives users high flexibility and speed for finding data sets in the large CDI catalogue and submitting their shopping requests. For the latter, users are asked to logon (or first register in case of a new user) with their Marine-ID at the start, like regular customers at e-shops. This gives access to MySeaDataNet services such as checking standing and historic orders, downloading data packages, and saving and re-using search profiles. This way the current separate RSM service for following up shopping requests has been integrated in the new CDI user interface. And being already logged on, users can submit shopping requests without further need for logging on again. A prototype of the new user interface has been demonstrated to and tested by the participants of the SeaDataCloud training workshops since June 2018 and their findings and suggestions have been evaluated and taken on board for refining.

Figure: Illustration of the new CDI user interface

For data centres the MySeaDataNet environment gives access to an online CDI Import Manager service with a dashboard to manage imports of new and updated CDI and data entries. And it gives data centres the option to evaluate and process shopping requests for restricted data sets, which are managed locally and not in the central data cache. The import of metadata and unrestricted data as well as the delivery of restricted data sets, once agreed, are structured as workflows and more self-service oriented as in the current situation. These services work together with the Replication Manager for efficient automatic processing. The Replication Manager replaces the Download Manager, and is easier to install and configure if done by technicians who are well aware of their local infrastructure. Alternatively, the semi-automatic ‘interim solution’ is maintained for data centres that are not allowed to install a local component. However, extra functionality is added to the CDI Import Manager for the CDI support desk to mediate for ‘interim’ data centres to maintain the CDI catalogue, now both with unrestricted and restricted data sets. This gives ‘interim’ data centres the option to relax their data policies and to provide unrestricted data sets. While moving from interim to a situation with Replication Manager is an option which adds to further streamlining the processes.

Figure: Architecture of new CDI service components with EUDAT part highlighted

The new CDI infrastructure is ready for replacing the current infrastructure and the migration process has started early June 2019. This is a challenging operation as the current CDI service should stay operational till the switch is made to the new service set-up and because also all related interfaces such as for EMODnet lots, Geo-Seas, BlackSeaScene, and other portals must be upgraded in sync with the SeaDataNet portal. It is also challenging because > 110 data centres have to migrate to the new configuration in a limited time period.

The migration process is coordinated by the CDI support desks at MARIS and IFREMER, who also provide the necessary support, guidance and testing for the more than 110 data centres to adopt the new configuration. More than 90 connected data centres with Download Manager have been requested to install and configure in parallel a connection with Replication Manager in TEST mode, while the existing Download Manager should continue to function in production mode to sustain public services. Each of these data centres are guided on an individual basis by the CDI support desk to implement the connection and to perform together an acceptance test by importing a test set of CDI metadata and related data sets. While data centres with ‘interim solution’ are asked to consider changing their data policy for (part of) their data sets from restricted to unrestricted and then to set the following steps together with the CDI support desk and/or to upgrade their configuration from interim to Replication Manager. Benefits of the Replication Manager solution include that all unrestricted data files are checked centrally for syntax and semantics conformance, while shopping requests can be served more efficiently than by semi-automatic arrangement.

Latest status of SeaDataNet tools

SeaDataNet has developed and maintains a set of tools to be used by each data centre and freely available from the SeaDataNet portal. It includes documentation and common software tools for metadata and data, statistical analysis and grid interpolation and a versatile software package for data analysis, QA-QC and presentation. As part of the SeaDataCloud project upgrades are undertaken taking into account new requirements. The following software versions are current:

MIKADO, developed by IFREMER, is used to generate the XML metadata entries for CDI, CSR, EDMED, EDMERP and EDIOS SeaDataNet catalogues. The latest version (3.5.2) has been released in January 2019, with only correction of few small bugs.

NEMO, developed by IFREMER, enables conversion of ASCII files of vertical profiles, time series or trajectories to SeaDataNet format files which can be either text files at Ocean Data View (ODV) and MedAtlas formats or binary files at NetCDF format. The latest version (1.6.7) has been released in January 2019 with some small improvements and bug corrections. The next version of NEMO (1.7.0) is under development, it will include the conversion to additional formats specifically designed for Biological, Microlitter and Flow cytometry data.

EndsAndBends, developed by IFREMER, is used to generate spatial objects from raw navigation (ship routes). Typical navigation log files record more than one location / 10 seconds (ex: GPS outputs) and the Size of these navigation log files are not practical to be managed or visualized using standard GIS software or services (WMS, WFS and GML). EndsAndBends enables a sub-setting of the navigation files, keeping the same geographical shape of the vessel route and reducing significantly the number of geographical locations to preserve response time.The latest version of EndsAndBends (2.1.0) has been released on April 2014, since then no new version has been developed.

SeaDataNet Reduce service, under development by MARIS, is an online service with comparable functionality as the EndsAndBends stand-alone software. It is being developed in response to data providers that want to reduce their navigation and survey tracks in batch for inclusion in their CDI metadata entries. The Reduce service expects GML input and makes use of fuzzy logic to determine the best fit reduction factor. It also gives map visualisation to compare the original track versus reduced tracks with different factors, including the advised version. The service is at present under testing.

OCTOPUS, developed by IFREMER, is a SeaDataNet format conversion software and also a format checker for SeaDataNet ODV, netCDF and MedAtlas files. The latest version (1.5.0) has been released in June 2019 and includes the checker for additional formats specifically designed for Biological, Microlitter and Flow cytometry data and some bug corrections. All data providers are strongly advised to use OCTOPUS to check their ODV, netCDF and MedAtlas files before initiating submissions in order to stick carefully to the agreed formats and rules. OCTOPUS includes already more than 500 checks.

Replication Manager (RM), developed by IFREMER, handles the replication of the local data sets managed in a SeaDataNet data centre into the central cloud. First public version of the replication manager (1.0.31) has been realised in June 2019 for the connection of all data centres. The Replication Manager is a web application which allows data providers to submit new and updated CDI files to the new CDI import manager at MARIS and at the same time to publish the corresponding unrestricted data files in the cloud for user downloading. The restricted data files are kept locally, in a specific directory, for authorised downloading.

Download Manager (DM), developed by IFREMER, supports connecting to the current SeaDataNet infrastructure. Since the last newsletter, there was no new release of the Download Manager, as it will be replaced by the Replication Manager in all data centres. The Download Manager will not be supported anymore as the upgraded CDI service becomes operational.

Training Workshops for uptake of new SeaDataNet standards, tools, and services

As part of the upgrading of the SeaDataNet infrastructure and capacity building for its data centres, 2 training workshops have been organised to date with as main target data managers and technicians of all the connected data centres. The SeaDataCloud project has a budget provision for allowing data centres, connected to SeaDataNet but not participating in the SeaDataCloud project, to join by covering their travel and daily subsistence costs. For the operation and uptake of new services, it is important that all connected data centres master the new developments improving and enriching the SeaDataNet services. On the one hand because they will have to use and/or implement the new features, on the other hand because they are on the front line for informing their community of users and customers of the new features available for a better and improved « SeaDataNet experience ».

The first training workshop was organized in 2 similar sessions from 20 to 22 June 2018 and from 25 to 27 June 2018 and with the support of the IOC-IODE at the UNESCO/IOC Project Office for IODE in Ostend, Belgium. This focused on standards and tools for populating the SeaDataNet European directories and quality control of several thematic data sets. Also, it gave a preview of the new CDI user interface for engaging the data centres in testing and providing feedback about possible problems and suggestions. This feedback has been used for further developing and finetuning the new CDI user interface which is planned to be publicly launched in August 2019.

Recently the second training workshop took place, again in 2 similar sessions in order to manage the large number of participants and to allow giving sufficient attention to them during the hands-on trainings. The sessions took place at the Thermae Palace in Ostend, Belgium from 19 to 21 June 2019 respectively from 24 to 26 June 2019. The sessions were co-organized by RBINS and IOC-IODE and welcomed 92 participants on site (51 for session 1 and 48 for session 2), including 15 trainers, while 13 participants joined remotely. Participation included the vast majority of the SeaDataCloud data centres (41 out of 43) and a great number of non-SeaDataCloud data centres (18 out of 76).

This year, the first two days of the workshop focused on the prototype of SeaDataNet Virtual Research Environment (VRE) and the VRE components: WebODV, DIVA online, Mapping services, and subsetting services (using ERDDAP). For the WebODV (Ocean Data View) and DIVA online (Data-Interpolating Variational Analysis) hands-on training exercises were undertaken. Finally an outlook was presented on the advantages of cloud-based research and further developments and wider uptake as envisioned for the VRE in SeaDataCloud, EMODnet projects, and the new Pilot Blue Cloud project. The third day focused mainly on the components of the upgraded CDI service, both the front and back ends, and the uptake by each of the data centres for supporting the migration to the new CDI service configuration which started early June 2019 and should be performed during July 2019, which is a major challenge considering the large number of data centre connections.

All presentations from both training workshops are published at the SeaDataNet portal and at the IODE OceanTeacher system. Moreover videos have been made of the presentations by IODE which are readily available for the first training workshop in OceanTeacher and for the second training workshop will be made available in OceanTeacher.

Reflection on IMDIS 2018 Conference and looking ahead to next IMDIS 2020 Conference

SeaDataNet initiated around 2005 the IMDIS conference - International Conference on Marine Data and Information Systems. The IMDIS cycle of conferences has the aim of providing an overview of the existing information systems to serve different users in ocean science. It also shows the progress on development of efficient infrastructures for managing large and diverse data sets, standards, interoperable information systems, services and tools for education.

The 6th edition of the International conference on Marine Data and Information Systems (IMDIS 2018) was hosted in Barcelona, by CSIC (Consejo Superior de Investigaciones Cientificas, Spain). It was co-organised by CSIC and IFREMER jointly with OGS (for the proceeding publication) and IOC/IODE (for the video recording and publication on Ocean Teacher) in the frame of the SeaDataCloud project.

The conference was held from 5 to 7 November 2018 in the auditorium of the PRBB (Barcelona Biomedical Research Park) on the seafront and next to the CSIC office.

Figure: The PRBB in Barcelona

The organisers were pleased to welcome 184 participants from 34 countries, mostly from Europe, but also from Australia, Canada, Cayman islands, Japan, Republic of Korea, and USA.

Figure: Number of participants per country

After a general introduction lead by the CSIC host and the SeaDataCloud coordinators, the conference was organised into 4 successive sessions:

1. Data services and tools in ocean science (16 oral presentations, 34 posters)
2. Technical developments for marine information and data management (11 oral presentations, 17 posters)
3. Marine environmental infrastructures for observation data (data management and access (9 oral presentations, 18 posters)
4. Data products, information and knowledge ( 17 oral presentations, 25 posters)

A total of 54 oral presentations and 96 posters were presented (pdf versions and videos of all presentations and posters are available on IMDIS web site.

The proceedings have been published in the Vol. 59 – supplement of Bollettino of Geofisica. You can also download an electronic version of the book of abstracts.

The next IMDIS 2020 Conference will also be organised in the frame of the SeaDataCloud project and it is planned for October 2020. On short term more details on the exact dates and call for abstracts will be published. IMDIS 2020 will be located in Amsterdam – The Netherlands and it will be locally organised by the Royal Netherlands Institute for Sea Research (NIOZ) together with MARIS.

SeaDataNet user workshops

During the last EGU General Assembly in Vienna on 11th April 2019 a SeaDataNet user workshop was organized as a Splinter meeting with title: “SeaDataCloud: useful tools and services for supporting oceanographic research and improving the transition from data to knowledge”. At the workshop different components of the SeaDataNet infrastructure were presented: metadata and data services, software tools (ODV and DIVA), and prototype Virtual Research Environment (VRE). The overall goal was to promote to the marine scientific community SeaDataNet with its services, standards, tools, data resources, data products and prototype collaborative workspace as very useful infrastructure for supporting their research.

The programme of the workshop was as follows:

• Introduction (Matteo Vinci, OGS)
• Recent developments of discovery interface and access services (Peter Thijsse - MARIS)
• Bringing the Ocean Data View (ODV) software to the Web (Sebastian Mieruch-Schnülle - Alfred Wegener Institute)
• Data-Interpolating Variational Analysis (DIVA) software tool (Charles Troupin - University of Liege GHER)

This way an overview was given to potential scientific users of the SeaDataNet services and SeaDataCloud developments for improving and expanding these services.

SeaDataNet established as legal entity

The general assembly of SeaDataNet II project, held in Split in September 2014, concluded on the necessity of consolidating the SeaDataNet network, on a more formal basis than a project consortium. As a first step, it was decided that an exploitation agreement had to be signed, by which each partner guarantees to keep the SeaDataNet infrastructure and services ongoing for at least 3 years after project end in order to bridge possible gaps in funding. In the weeks that followed 43 partners signed the agreement.
In parallel, work begun to establish SeaDataNet as a legal entity, as a non–profit association under the Belgian legal regime. After a lot of preparatory work, involving the legal department of all the parties involved (and, hence, sometimes, long delays before getting an agreement on an amendment to the proposed statutes…), all the administrative steps have been fulfilled. The Statutes of Association Internationale Sans But Lucratif SeaDataNet (AISBL SeaDataNet) have been published in the Annex of the Belgian official journal (“Moniteur Belge, Annexe des personnes morales”) and became legally effective on May 5th, 2019. Its registered office is located at the Royal Belgian Institute of Natural Sciences, rue Vautier 29, 1000 Brussels, Belgium.
The association aims to work in the collective interest of its members to improve the quality and effectiveness in the management, the processing and the dissemination of oceanographic data at national, regional and global levels. More specifically the purpose of SeaDataNet AISBL is to:

• Promote the SeaDataNet infrastructure, tools and services and the development of underpinning science and technology;
• Foster cooperation and coordination within its Members in the field of data management, processing and dissemination;
• Promote and coordinate the development of commonly available products and services; and
• Coordinate the contribution of the pan-European oceanographic data management community to the European Union initiatives for a better environmental status of the seas and oceans and for an improved marine knowledge in general.

The members of the executive board of directors are: Mrs Michèle FICHAUT (Vice–Chair), Mrs Alessandra GIORGETTI, Mrs Athanasia IONA, Mrs Lesley RICKARDS, Mr Dick SCHAAP, Mr Serge SCORY (Chair). Their mandate ends at the General Assembly of the association in 2022.

There are two possible mechanisms for becoming a member:

• Organizations, who are signatories of the SeaDataNet Exploitation Agreement shall be automatically admitted as a Member upon notification of their written request for affiliation addressed to the Organization and after reception of their annual membership fee (250€).
• Otherwise, new Members must be admitted by the General Meeting. Admission guidelines and criteria are set out in the Internal Rules, it being understood that at least the (corporate) purpose of the candidate member should be compatible with the purpose of the Organization.

In a near future, further information (application letter template, bank account) will be sent to all the organizations falling into the first category; while also information will be provided on the SeaDataNet website for potential other organisations seeking to become Member.

Memorandum of Understanding with CMEMS

In June 2018 the existing Memorandum of Understanding between SeaDataNet (SDN) and Copernicus Marine Environmental Monitoring Service (CMEMS) has been amended and renewed. Considering their respective activities and roles, CMEMS and SDN recognize the fact that their in-situ data management activities are complementary and that there are several areas of development and operational activities that will benefit from cooperation between them. Therefore, CMEMS and SDN have agreed to cooperate in strategy, research, and operation, where mutually beneficial and desirable. The Memorandum of Understanding formulates an initial number of areas and activities of cooperation, while this list can be updated over time in mutual agreement.

• Development, maintenance, adoption and promotion of standards
• Development, adoption and promotion of QA-QC methodologies
• Metadata and data exchange from SDN to CMEMS INSTAC
• Metadata and data exchange from CMEMS to SDN
• Development of products
• Distribution of products
• Teaming up in relevant projects

This Memorandum of Understanding has an indefinite duration and entered into force 15th June 2018 on the day of its signature by the signatories on behalf of CMEMS and SDN.

How to manage HF Radar data sets

High Frequency Radar (HFR) is the unique land-based remote sensing technology that allows the mapping of sea currents fields over wide areas with high spatial and temporal resolution. HFR products are applied in many sectors such as research in coastal oceanography and marine environment, safety, coastal management, fishery, navigation and renewable energy. Already identified as a relevant emerging technology in the framework of the Global Ocean Observation System (GOOS), HFR data will be systematically assimilated in models for forecast assessment and adjustment.

In the last decade, HFR technology has started rapidly expanding in Europe. The European HFR network, with over 62 HFR antennas currently operating and a number in the planning stage (Rubio et al, 2017) and with a growing rate of 7 new systems per year since 2016, is the second widest worldwide.

In 2014, EuroGOOS launched the HFR Task Team to achieve the harmonization of system requirements and design, data quality and standardization of HFR data access and tools. In 2015, a pilot action coordinated by EMODnet Physics, begun to develop a strategy for assembling HFR metadata and data products within Europe in a uniform way to make them easily accessible, and more interoperable. Recently, the Copernicus Marine Environment Monitoring Service (CMEMS) service evolution call supported the INCREASE project, which will set the basis for the integration of existing European HFR operational systems into the CMEMS-INSTAC (In Situ Thematic Assembly Center). In parallel, the EU project JERICO-NEXT is working on developing procedures and methodologies to enable HFR data to comply with the international standards regarding their quality and metadata, within the overall goal of integrating the European coastal observatories.

These efforts will facilitate the inclusion of HFR data into the major European platforms for marine data distribution, namely CMEMS-INSTAC, SeaDataNet, and EMODnet Physics. The operational Near Real Time (NRT) data stream from HFR is already included in EMODnet Physics since 2015 and in CMEMS-INSTAC since April 2019. For the validation and long-term preservation and distribution of historical HFR time series it was decided to adopt the SeaDataNet infrastructure. Therefore, activities are undertaken in the SeaDataCloud project for formulating standards for long-term archiving of HFR data.

Following the CMEMS-INSTAC requirements, a first version of the European common data and metadata model for NRT HFR-derived current data was defined and implemented in 2017, compliant with Climate and Forecast Metadata Convention version 1.6 (CF-1.6), OceanSITES convention, and INSPIRE directive. Furthermore, a list of Quality Control tests to be applied to NRT HFR current data was defined according to both the EuroGOOS DATAMEQ working group recommendations on real-time Quality Check procedures and the Quality Assurance/Quality Control of Real-Time Oceanographic Data (QARTOD) manual defined by the US Integrated Ocean Observing System (IOOS).

Starting from this standard data and metadata model, further work has been carried for compliance with SeaDataNet requirements. In 2018 this has resulted in formulating how historical HFR current data can be described for metadata in the SeaDataNet CDI format and for data in the NetCDF4 (CF) format for gridded data sets. The model, released in its final version in 2019, specifies the file format, the global attribute scheme, the dimensions, the coordinate, data and Quality Control (QC) variables and their syntax, the QC procedures and the flagging policy. The data model is described in the SeaDataCloud D9.12 deliverable.

Recognising the importance of HFR data, the European HFR Network Node (EU HFR NN) has been established which acts as the focal point with European HFR data providers, the key EU marine data distribution infrastructures and the Global HFR network. The EU HFR NN is responsible for supervising and operationally running the full HFR data processing and data management chain from data providers to the distribution services. Thus it guarantees the timely production and distribution of quality controlled NRT and historical datasets, with their associated metadata, towards CMEMS-INSTAC, SeaDataNet, and EMODnet Physics.

SEANOE service, publishing marine scientific data

SeaDataNet uses the SEANOE (SEA scieNtific Open data Edition) service to facilitate scientists to publish their research data in the field of marine sciences as citable resources. SEANOE has been developed by IFREMER and was launched as a DOI minting service in 2015. Recently a version of SEANOE has been adapted for SeaDataNet using the SeaDataNet template to be integrated in SeaDataNet website, as a new service called “Publish your data” and available from the SeaDataNet homepage.

 Figure: SEANOE as DOI minting service in SeaDataNet portal

Each data set published by SEANOE has a DOI. The data may well be cited in scientific articles in a reliable and sustainable way. Publishing in SEANOE is free and Data published by SEANOE are available for free. They can be used in accordance with the terms of the Creative Commons license selected by the author of the data. SEANOE contributes to Open Access / Open Science movement for a free access for everyone to all scientific data financed by public funds for the benefit of research. An embargo limited to 2 years on a set of data is possible; for example to restrict access to data of a publication under scientific review.

Figure: SEANOE Landing Page’s DOI example

SEANOE offers a solution for publishers that strongly recommend to cite datasets in articles. It is recommended by Elsevier, PLOS, … SEANOE offers a fast responding service: if a dataset is well described, it should get a DOI within 24 hours.

A survey of dataset DOI citations in articles is performed with Google Scholar. Article references are reported to authors, displayed on the dataset Landing page and the couples Dataset DOI / Article DOI are provided to DataCite.

The Landing Pages are optimized for a better visibility on Google and Google Dataset Search. Searches in Google are at the origin of most of the dataset downloads.

Figure: DOI’s Landing Page consultation in 2018

A selection of the data sets published in SEANOE are also automatically duplicated to the EMODnet Data Ingestion portal so that marine data centers of the EMODnet Ingestion network will be informed about the existence and the availability of datasets published in SEANOE. Those data centers will then undertake activities for elaborating submitted datasets for uptake in their national databases and wider publishing in European data portals such as SeaDataNet, EurOBIS, and EMODnet thematic portals, if the datasets are of interest for their portfolio.

Figure: Example of landing page in EMODnet Ingestion of data set publication taken over from SEANOE

Developing the SeaDataNet Virtual Research Environment (VRE)

The overall philosophy for the SeaDataCloud development is illustrated and explained as follows:

Figure: SeaDataCloud development philosophy

Providing a cloud platform with common services for data pre-processing, subsetting, analyses, visualizations, publishing, DOIs… Applying common standards and interoperability solutions for providing harmonised data and metadata Providing harmonised discovery and access to data output from multiple sources, such as European research and monitoring data gathering, but also from other European and international data infrastructures

The overall philosophy is to provide researchers access to an extensive and steadily increasing pool of available marine and ocean data resources from a large network of data providers, supported by standard, interoperable and highly efficient services for discovery, access, and transformation of data resources to common formats. Those data resources can then be applied by researchers for performing analytical operations as cloud-based science. For the latter SeaDataCloud is developing a SeaDataNet Virtual Research Environment (VRE) which hass now progressed to prototype level. It provides a set of advanced services for analysis, quality control, sub-setting, and visualisation of retrieved datasets and generation and publication of data products. Further integration is planned as well as try-out of the VRE prototype by the SeaDataCloud data products group for supporting generating (part of) a T&S climatology in a collaborative manner.

Figure: Set-up of the SeaDataCloud VRE prototype

Prototype VRE: WebODV – online data analysis

As a crucial part of the SeaDatanet VRE prototype, an online version of the ODV software is being developed named webODV. This provides ODV functionality via user-friendly web browser interfaces, which communicate via WebSockets with ODV instances running on the server, providing access to server based datasets. There is no need for local ODV software installation and the, sometimes very large, datasets do not need to be downloaded to the user’s system. Both resources are run and opened on the server. webODV can be used on many operating systems and devices (desktop, laptop, tablet, smartphone) as long as an Internet connection is available. This approach simplifies access to large community datasets, such as the aggregated SeaDataNet database and increases dissemination and usefulness of the data. webODV will be integrated in online workflows, such as quality control, and can be used for collaborative work. Additionally, it can be customized for special needs and can well support outreach, training and education activities, without demanding powerful end-user hardware.

webODV services are embedded in Docker containers, thus they are modular, flexible and easy to use and maintain on the SeaDataCloud servers. The following webODV services have reached a “beta” status and are expected to become operational at the end of September 2019.

Import / aggregation service:
A typical requirement of SeaDataCloud data users is the aggregation of many individual data files into a large and easy to use data collection. As yet, this step has been accomplished using the desktop version of ODV, now data aggregation can also be performed online in the VRE.

 
Figure: webODV online data import

As shown in the screenshots of the import service, the user has to go through 4 steps to aggregate data, i.e. selecting import data type, selecting the files to be imported, specifying an output directory and finally starting the import process. The aggregated data collections are automatically copied into the user’s workspace in the cloud.

Quality control service:
Quality control (QC) of the imported data is a very important step performed regularly by SDN regional experts. These experts inspect scatterplots of variables like temperature or salinity and identify outliers, which are then flagged according to the SeaDataNet flagging scheme. The following figure shows the online version for quality control in the VRE that provides a wide range of QC functionality. Like in the desktop ODV, users select stations or samples by left mouse clicks and obtain context menus via right clicks. The online QC capabilities match those of the desktop ODV version and work in the same way. Previous desktop ODV users will be productive from the beginning. All changes are logged.

Figure: webODV online quality control

Data extraction service:
The webODV data extraction service provides easy to use and fast access to the large SeaDataCloud aggregated datasets. As an alternative to full dataset downloads, for instance via the SeaDataNet data products catalogue, users can create personalized data subsets by applying a wide range of station and sample selection criteria and then download the smaller data subsets in various formats. As shown in the following figure, the user is guided by a 4 step pager to select stations and variables and finally download the data in different formats (e.g. .odv, .nc) into the cloud workspace.

Figure: webODV online data extraction

Prototype VRE: DIVA online for generating interpolated maps

DIVA (Data-Interpolating Variational Analysis) is one of the core software tools developed in the frame of the SeaDataNet - SeaDataCloud project suite. It has been designed to perform spatial interpolation of oceanographic observations in an efficient (i.e. dealing with up to a million of data points) and consistent (i.e. taking into account the physical boundaries) way.

Within SeaDataCloud, a new DIVA code has been written and developed in order to work in an arbitrary number of dimensions: DIVAnd, while the previous DIVA was only dealing with two-dimensional interpolations. DIVAnd is not a new release of DIVA, but a brand new software tool with a different mathematical formulation, a different solver and another programming language: Julia. Julia is a dynamically-typed, high-performance and open source language, whose development started in 2009. It perfectly suits our needs for solving efficiently problems involving large matrices.

Installing and using DIVAnd:
The tool requires the installation of Julia. It is distributed as a regular Julia package so its installation is seamless. The code is also distributed via GitHub, allowing users to get the development code, open issues and create pull-requests. Within the module, several examples explaining how to use the tool are provided. In addition, a set of jupyter notebooks is distributed both as training material and user guide for the creation of climatology and other gridded products.

DIVAnd and the SeaDataNet Virtual Research Environment prototype:
In order to provide users with the tools and the computing power, DIVAnd has been deployed in the Virtual Research Environment (VRE), thus making it possible to users to connect to the infrastructure, access comprehensive datasets and perform spatial interpolations on the cloud.

DIVAnd is deployed as a Docker container and used with a JupyterHub application, which provides notebooks to groups of users. A graphical user interface on top of a REST API allows users to easily configure and run analysis using their own dataset.

The complementarity with ODV is obvious: webODV is employed to prepare and quality-control a dataset, which is then fed into DIVAnd to produce a set of gridded fields.

SeaDataNet and the ENVRIFAIR project

ENVRI-FAIR is a new H2020 project which started in January 2019. It is the connection of the ESFRI Cluster of Environmental Research Infrastructures (ENVRI) to the European Open Science Cloud (EOSC).

Participating research infrastructures (RI) of the environmental domain cover the subdomains Atmosphere, Marine, Solid Earth and Biodiversity / Ecosystems and thus the Earth system in its full complexity. The overarching goal is that at the end of the proposed project, all participating RIs have improved their FAIRness and are ready for connection to EOSC. The Marine subdomain includes Euro-ARGO, ICOS Marine, EMSO, and Lifewatch Marine, while SeaDataNet is added as technical partner considering its expertise and experience with marine standards and interoperability.

The workpackage on achieving FAIRness for the Marine Subdomain is led by IFREMER, while MARIS leads the task on assessing and analyzing the FAIRness of each Marine RI and formulating a roadmap for the Marine subdomain improving the FAIRness. Moreover, SeaDataNet experts are involved in the workpackage concerning horizontal tuning of FAIRness activities between the 4 subdomains and seeking synergy in developing and adopting common solutions, where possible, for improving FAIRness.

FAIR focuses on “Data and services that are Findable, Accessible, Interoperable, and Re-usable, both for machines and for people.” The FAIR movement started with an article in Nature in 2016:

Figure: Article in Nature about FAIR guiding principles

The FAIR guiding principles are mostly directed towards metadata.

Further work on FAIR is undertaken by GOFAIR and the Research Data Alliance (RDA). As part of ENVRIFAIR use is made of a questionnaire developed by GOFAIR for analyzing the FAIRness of each of the RI’s, including SeaDataNet. This is followed by a strengths and weaknesses analysis, resulting in plans for improving weaknesses. For SeaDataNet most effort will be focused on improving Reusability as F, A, and I are already quite well developed and underway. Solutions will be found in synergy with SeaDataCloud developments such as enriching CDI metadata, for example by making optimal use of the Linked Data activities, while other options might be identified from the cooperation and communication with the other RI’s in ENVRIFAIR.

EMODnet developments with SeaDataNet involvement

The European Marine Observation and Data network (EMODnet) was initiated in 2008 by EU DG MARE and from the start SeaDataNet members have been closely involved in its planning and implementation. SeaDataNet infrastructure and standards have been adopted for developing and supporting the EMODnet portals for physics, chemistry and bathymetry, while also contributing to biology and geology. The focus in EMODnet is on developing generic data products and serving these with effective interfaces and services to users from public, government, research and industry.

These EMODnet projects and portals make full use of the SeaDataNet infrastructure for managing additional thematic measurement data as gathered from data partners, connecting new data providers to the SeaDataNet data discovery and access service, and for harvesting large data collections from the SeaDataNet infrastructure which serve as input for the production and publication of thematic data products with a European coverage.

In addition, SeaDataNet is active network of data centres in EMODnet Data Ingestion, charged with seeking elaboration of ingested data sets to common standards and uptake of these resulting data sets in the SeaDataNet infrastructure.

The third phase of EMODnet has been underway since early 2017 and is nearly finalized with many new results. In the meantime, already several contracts for EMODnet projects have been continued for another 2 years, while others are waiting for the outcome of a Call for Tender that was closed in April 2019 for continuing services and activities for another 2 years and possibly 4 years. In the following articles a summary of the results is given for the EMODnet projects with SeaDataNet involvement.

Update on EMODnet Chemistry

EMODnet Chemistry phase III ran from March 2017 till May 2019 and the partnership included 45 institutes from 27 countries and 3 international organisations (ICES, Black Sea Commission, UNEP/MAP) with overall coordination by OGS (Italy) and technical coordination by MARIS (Netherlands). The scope included to continue and strengthen the collecting, validating, and providing access to marine chemistry data, and generating and publishing marine chemistry data products, relevant for the implementation of the Marine Strategy Framework Directive (MSFD) and its stakeholders at national, regional and European levels. Focus was on MSFD indicators 5 (Eutrophication), 8 (Contaminants) and 9 (Sea-food contaminants) engaging and tuning with major MSFD stakeholders. This scope was extended with gathering data and generating products for indicator 10 (Marine Litter), in particular beach litter, seafloor litter as found in fishermen’s nets, and micro plastics. This has been undertaken in cooperation and synergy with existing initiatives at regional and European scales, including tuning with the MSFD Technical Group on Marine Litter (TG-ML). Also the geographical scope has been broadened, covering all major European sea regions: Norwegian Sea and Barents Sea, Baltic Sea, N.E. Atlantic (Celtic Seas, Iberian coast and Bay of Biscay and Macaronesia), Greater North Sea, Mediterranean Sea and Black Sea, thereby adopting the geographical region definitions of MSFD.

The EMODnet Chemistry portal has been completely redesigned in styling, texts, and sitemap in order to put the services for discovery and access of marine chemistry data and data products in a central position. For users it is now directly clear that the EMODnet Chemistry portal is dealing with data themes of relevance for MSFD, namely Eutrophication, Pollution, and Marine Litter.

Figure: Restyled homepage of EMODnet Chemistry portal

This included upgrading and reviewing of the various services at the portal:

• CDI Data Discovery and Access Service by means of a CDI matrix of chemical groups per MSFD sea region. And the standard Chemistry CDI user interfaces for extended search and quick (facet) search have been upgraded giving a more clear focus on chemical substance groups and the 3 matrices (water, sediment, biota).

Figure: CDI Matrix of number of CDIs per chemical group per MSFD region; clicking on cell performs a dedicated query on the ‘regular’ CDI user interface for giving CDI results

• OceanBrowser Viewing Service: giving facilities for viewing, browsing and downloading Chemistry aggregated validated and harmonised data collections for eutrophication and contaminants and DIVA interpolated sea basin maps for eutrophication.

Figure: OceanBrowser service displaying interpolated seabasin map for water body phosphate

• Sextant Products catalogue service, giving facilities for searching and downloading Chemistry data products through the link with the OceanBrowser viewing service. Each data product has a DOI and related landing page and gives acknowledgement to data providers.
• Advanced viewing services for time series and profiles, allowing users to generate and view dynamic plots for selected parameters from data sets, selected from the harmonised, aggregated and validated data collections.

Data collection and metadata population:

The gathering of data concerning eutrophication (nutrients, chlorophyll and oxygen) and selected contaminants has been continued and together with input from non-EMODnet Chemistry data providers the number of relevant entries in the dedicated Chemistry CDI service has increased from 841356 in March 2017 to 999797 records end April 2019, provided by 64 data centres from 32 countries.

Figure: Example of the EMODnet Chemistry CDI Data Discovery and Access service

Marine litter was a new topic in EMODnet Chemistry with a focus on beach litter (nets, bottles etc.), seafloor litter (i.e. litter collected by fish trawl surveys), and floating micro plastics. In dialogue with the MSFD Technical Working on Marine Litter (TG-ML) and other stakeholders it was decided to specify and set-up two central EMODnet databases for managing beach litter respectively seafloor litter. The database for beach litter has been modelled following the OSPAR-MCS approach and takes into account TG-ML and UNEP/MAP requirements. The database on seafloor litter is modelled following the ICES-DATRAS approach and takes into account TG-ML and MEDITS requirements. The database for beach litter was populated by harvesting from the OSPAR – MCS database, together with Member States data contributions for years from 2012 to 2016, as part of an intense cooperation with JRC and their EU project on Marine Litter baselines, and additional datasets from the partnership. The database for seafloor litter has been populated by harvesting from the ICES-DATRAS (the Database of Trawl Surveys), which includes also Baltic International Trawl Surveys (BITS) data, used for national fish trawl litter. Further communication is ongoing with the Mediterranean community, in particular MEDITS (DCF related surveys) and DeFishGear (EU project), for arranging their input for the EMODnet seafloor database, also on a regular interval. EMODnet Chemistry is supported by TG-ML and JRC in this. Dedicated Guidelines and Forms for gathering beach litter and seafloor litter were made and are published at the EMODnet Chemistry portal and submitted to the Ocean Best Practices web site. The collected ML data in the two central databases have also been described as CDI metadata and have been populated into the CDI service to facilitate easy discovery by users.

For micro plastics, the data management situation was less consolidated and more fragmented so it was decided to adopt and adapt the SeaDataNet CDI and ODV standards. A dedicated Guideline for describing marine micro-litter data sets was published after consultation with TG-ML, JRC, and the JPI-Oceans BASEMAN project. As part of the ML approach also new vocabularies for ML have been defined and these have been added to the overall SeaDataNet vocabularies.

Generating and publishing data products:

The collected data sets for eutrophication, contaminants, and marine litter have provided input for regional groups to generate a series of data products:

• Harmonised and validated eutrophication aggregated datasets (version 2018) for 6 marine basins;
• Interpolated maps on parameters related to eutrophication, available for 6 marine basins. Maps are available at seasonal basis, as moving 6-year analysis;
• Harmonised and validated contaminants aggregated datasets (version 2018) for 6 marine basins;
• Dedicated maps on contaminants;
• Validated maps for marine litter.

Figure: European beach litter map of mean number of plastic bags related items per 100 m

Figure: European beach litter map of mean total of beach litter items per 100 m

The products are described with metadata in the Chemistry products catalogue, have DOIs and landing pages for citation, and can be viewed in the OceanBrowser service, while there are also web services for sharing with other portals.

Update on EMODnet Bathymetry

The EMODnet High Resolution Seabed Mapping project ran 20th December 2016 till 19th December 2018. It was a follow-up of the earlier developments in EMODnet Hydrography, Seabed Mapping, and Bathymetry projects which took place since June 2009 and that resulted in the versatile EMODnet Bathymetry portal. The partnership was expanded from 24 to 41 active members while being coordinated by Shom (France) with MARIS (Netherlands) as technical coordinator.

The portal and services have been outfitted with an upgraded design and layout, responsive design, and conforms to the new overall EMODnet style.

Figure: Homepage of EMODnet Bathymetry portal

The first part of the project was dedicated to training and supporting data providers in populating the CDI data discovery and access catalogue with bathymetric survey entries and the Sextant catalogue with composite DTM entries. The total number of CDIs has increased over the project period from 14791 to 27168 records and Composite DTM entries from 78 to 147. The latter also includes 18 satellite derived Composite DTMs for the Mediterranean coastal zones of Spain, Greece and Libya. The number of data providers has increased from 28 to 51.

The methodology, software tools, and guidelines for gathering and processing bathymetric survey and composite DTM data sets from data providers into Regional DTMs were improved. The GLOBE software for pre-processing and pre-gridding of input data sets and generation of the Regional DTMs was upgraded. Not only regional coordinators but also data providers were encouraged to use it preparing their data input. This way there was a harmonized process in all steps from data input to Regional DTMs.

Image: Workflow for EMODnet HRSM

24th September 2018 a new and upgraded version of the EMODnet DTM (Digital Terrain Model) was officially released. Compared to the 2016 release, grid resolution has improved from 1/8 * 1/8 to 1/16 * 1/16 arc minutes (circa 115 * 115 m2). It contains approx. 12.3 billion grid nodes, organized in 113892 columns and 108132 rows (seabed and terrestrial coverage included) while e.g. GEBCO has 933 million grid nodes for worldwide coverage. From all the data sources gathered, a total of 9369 unique CDI references and 87 DTM references are used in the overall DTM (6924 in 2016 version). It is available free of charge for viewing and downloading, and sharing by OGC web services from the EMODnet Bathymetry portal.

Figure: New EMODnet DTM with higher resolution and including arctic waters

Other new features of the new release are:

• A powerful 3D bathymetry visualisation functionality in the viewer without plugins;/li>
• All European seas including part of the Arctic Ocean and Barents Sea;
• Inclusion of Satellite Derived Bathymetry data products, in particular for coastal stretches of Spain and Greece;
• Improved source reference layer with quality indication: the source reference layer gives a map with polygons which correspond to contributing surveys and composite DTMs for which related metadata can be retrieved. Metadata has been expanded with characterization of the dataset by vertical, horizontal and temporal indicators, purpose of the survey, and information about commonly adopted standards. Analysis resulted in extra maps with Quality Indicators for each source reference, which have been integrated in the Bathymetry viewer as extra layer;
• Downloading of DTM tiles is integrated into a shopping mechanism which facilitates registration of users and their reasons for use.

 
Figure: View from the South along Sicily and the Southern part of mainland Italy

In addition, several other new products were generated and published:

• An inventory and report presenting baseline and coastline data as collected from 21 national authorities. This can be downloaded from the portal together with shapefiles of the baselines and coastlines;
• Best-estimate coastlines were determined from satellite data (typically Sentinel-2 and Landsat-8) and in combination with the Global Tide Surge Model (GTSM) processed into digital coastlines for the European seas at LAT (Lowest Astronomical Tide), MSL (Mean-Sea-Level), and MHW (Mean-High-Water). Coastlines can be viewed as extra layer in the Bathymetry Viewer and downloaded with documentation from the EMODnet Bathymetry portal;

 
Figure: Water occurrence estimated for Wadden Sea from multiple satellite images

• Using the GTSM DTM tiles of the 2018 Version can be downloaded both relative to LAT and MSL reference levels;
• A multi-resolution layer has been added consisting of a collection of even higher resolution composite DTMs for selected areas which have been gathered and processed by data providers into High Resolution DTM (HR-DTM) files. The resolution of HR-DTMs varies between 1/32 and 1/512 arc minutes, depending on local data policy of data providers. All HR-DTMs are described with metadata in a special section of the Sextant Catalogue. In total 196 HR-DTM files have been provided can be viewed, interrogated for metadata and downloaded by an extra HR-DTM layer in the Bathymetry viewer.

Figure: Example of HR-DTM: Vulcanic seabed surrounding the Greek island of Santorini

The GIS layers in the Bathymetry Viewing and Download service can be shared as OGC WMS and WCS services with other EMODnet portals and beyond. Also, WMS layers from other EMODnet portals and external services can be added to the Bathymetry Viewer and Download service. The URLs for the OGC services can be found at the portal.

A new phase for EMODnet Bathymetry started in December 2018 for another 2 years and is named "EMODnet Bathymetry - High Resolution Seabed Mapping’. It is a continuation of the previous project with the same partnership and comparable activities with the overall aim to refine the EMODnet DTM further with additional high-quality data sets and improved methodologies.

The portal and the EMODnet DTM are very successful with many users from research, government and industry. EMODnet Bathymetry is cited on multiple occasions as a successful regional project involving both hydrographic offices and research institutes with complementing approaches in terms of data coverage and methodologies (acquisition, processing and validation). The Seabed 2030 initiative (https://seabed2030.gebco.net/), led by the IHO-IOC GEBCO, under the financial sponsorships of the Nippon Foundation and launched on the 6/6/2017, recognised EMODnet Bathymetry as a worldwide key actor of bathymetric data production (https://seabed2030.gebco.net/documents/seabed_2030_roadmap_v10_low.pdf). The web portal and its services are well visited with > 10.000 unique visitors per month. The OGC web services (machine-to-machine) are very popular with more than 250.000 visitors per year. The number of downloaded DTM tiles amounts to circa 10.000 per 3 months. With the launch of the new DTM end September 2018 registration was added of the identity of downloaders and reasons for use. In the last quarter of 2018 the new service registered > 12.000 downloaded tiles from circa 1700 unique users from > 800 organisations from industry (22 %), government (6%), research institutes (18%), universities (43%), public (8%), international organisations (1%) and NGO’s (2%). Thereby each user indicated trustworthy reasons for use.

Update on EMODnet Physics

The main goal of EMODnet Physics is to provide data and products for specific operational systems and for specific maritime economic activities. More specifically, the users of EMODnet data are asking for some key actions:

• Improve the extent, completeness and ease of access to marine data required by industries and agencies supplying products to industry (e.g. environmental and productivity information for sea- culture and fishing, pollution and operational weather forecasting for offshore energy and mining) and citizens directly (e.g. weather forecasts, water quality predictions, marine food safety). The improvement in quality and access to data will reduce costs and delays in existing industries and promote the development of emerging markets and practices (e.g. seabed mining, offshore aquaculture);
• Implement more cost-effective multi-use platforms with improved capability and durability, measuring synchronously more interlinked variables (physics, biogeochemistry and biology), improving accurate and timely data delivery, and by systematically filling observational gaps for specific under-sampled areas;
• Harmonize data processing and quality control procedures;
• Close the gap between continental shelf and Deep Ocean observing networks and close the gap between near real time observation delivery and its high-level qualification and long-term preservation;
• Promote a free-share of observing data at any level (local, national, European and international).

Activities such as the construction of ports and marinas, shipping and ferry services, offshore infrastructures, marine and costal protection need data on currents, waves, atmospheric conditions, sea level, temperature and salinity, pH, turbidity, river runoff, and even ice coverage in northern seas. During the last decades, ‘noise’ in the marine environment has been recognised as one of the issues of major significance. EMODnet Physics is starting to collect data to provide information on temporal and spatial distribution of acoustic disturbances.

Data and products accessible through the EMODnet Physics portal can also support the European Marine Strategy Framework Directive and Water Framework Directive by providing data useful for acidification, mixing characteristics, residence time, upwelling, wave exposure. This is illustrated by the following tables:

EMODnet Physics portal:
The EMODnet Physics portal is providing a single point of access to near real time data, delayed data and reprocessed data. Data is updated three times a day. Near real time (NRT), datasets are managed in a cooperation with the EuroGOOS ROOSs and the CMEMS INS TAC. Regional Data Assembly Centre (RDAC) are constituting decentralised assembling systems connecting organisations operating monitoring stations. Harmonised QA/QC procedures assure the quality of the products delivered to users. EMODnet Physics Portal makes available near-real-time data for past 60 days in daily datasets. Older data are made available in “monthly” dataset (every month the latest 30 days data are reorganized into the “monthly” dataset file). Each platform can provide one or more parameters.

Figure: Homepage of the EMODnet Physics portal

Historical validated data is organized in coordination and cooperation with SeaDataNet and the network of National Oceanographic Data Centres (NODCs). Validated historical datasets are made available via the EMOdnet Physics backoffice infrastructures. The access to validated archived data sets is arranged by means of the SeaDataNet Common Data Index Data (CDI) Access and Discovery service.

Data acquired by networks such as ARGO and OceanSites are also made accessible through the EMODnet Physics portal, by mutual agreements. The most challenging activity for EMODnet Physics is to cooperate with these networks in order to have a better exchange of metadata and establish a more sustained dataflow from the identified national or international data repositories. When a dataset is present in multiple repositories, it is important for EMODnet Physics to provide the user with the proper information on the original repository.

Although EMODnet Physics is supporting an open and unrestricted data access, some data older than 60 days may require user identification. Common QA/QC protocols as well as best practices have been collected and made available through the page http://www.emodnet-physics.eu/portal/bibliography. On top of these common standards, EMODnet Physics develops and provides a further level of interoperability tools such as WMS, WFS, web services in order to make these data accessible, discoverable and usable by a wider community.

Conclusion:
EMODnet Physics does not operate in situ observing systems but collects observations in cooperation and coordination with EuroGOOS ROOSs, CMEMS INSTAC and SeaDataNet network of National Oceanographic Data Centres. These are the networks developing, adopting and adapting the best available standards and procedures for metadata and data flow. The data flow includes the data owners/providers, the regional and inter-regional integration infrastructures, standardization and quality control and open access and free access for latest 60 days of near real time data. Data from international programs (e.g. ARGO) are free and open without any time window limitation.

The EMODnet Physics outputs are a source of timely and up-to-date information about environment and security for the benefit of individual citizens and decision-makers not only in Europe but also on a global scale. The portal is developed on social, environmental and economic issues that need to have a single point data and information in order to analyse them with a holistic approach.

One positive aspect of the EMODnet Physics programme has been the capacity to involve environmental agencies and groups working at national and regional level (e.g. the regional Conventions), as well as institutions from United States and Australia. An important action coordinated within EMODnet Physics was the free, open and easy access to data, products and information in a federated interoperable environment.

The result has been a consolidated partnership and synergetic exchange of resources. Responsibilities are equally distributed and coordination is very effective, with the main result of data provision in a timely manner.

Update on EMODnet Biology

During the last year, EMODnet Biology has achieved several milestones. The number of dataset and occurrence records has grown. Since the beginning of Phase III, 149 datasets have been added or updated, accounting for 3,902,811 new occurrence records from different functional groups and covering all the European sea basins. The temporal, functional and geographical distribution of the new records is represented in the chart and map below. A good proportion of the new data occurrences are benthos and phytoplankton data, and have been collected in the last 20 years in the Greater North Sea, Baltic Sea, Mediterranean and Arctic waters.

Figure:. Number of new occurrence records added in since the beginning of Phase III of EMODnet Biology, dusplayed per year of observation and functional group.

Figure: Geographical distribution of the occurrence data (gridded) added in EMODnet Biology since the beginning of Phase III.

Another important milestone was the publication of the European Atlas of Marine Life in December 2018. The Atlas presents a range of data products including tools, models and maps, illustrating the diverse range of outputs that can be generated from EMODnet Biology hosted data. The available data and products are structured around the biological Essential Ocean Variables (EOVs) facilitating global interoperability and contributing to the aims of UN Sustainable Development Goal 14, the relevant Aichi Biodiversity Targets of the Convention on Biological Diversity and the activities focused around the UN Decade of Ocean Science (2021-2030). All underlying data is open and the code and methodologies used in the creation of the data products are available in the EMODnet shared software repository.

Figure:One of the products of the European Atlas of Marine Life: “Distribution of benthic macroinvertebrate living modes in European seas”. It consists of a series of products displays the main functional types of benthic macroinvertebrates derived from a multivariate analysis of 13 life history traits.

Last February, EMODnet Biology launched a new version of the download toolbox. The new features include the possibility to select records based on data precision (taxonomic rank, date precision and/or coordinate uncertainty). Users can also now interrogate the database to select data with additional parameter filters based on controlled BODC vocabularies: (e.g. extra data on biota descriptors & quantifications, currents, sea level and waves, dissolved gases, bathymetry, meteorology, optical properties, sediment properties, sampling descriptors or water column temperature and salinity, or FCM data). In the last step, users can choose between different download types and visualize their final selection on the geoviewer. Users will receive an email with a link to the location where the dataset is available for download.

Figure: Screen capture of the step in the EMODnet Biology download toolbox where users can select data based on additional data standardized to BODC vocabularies.

Besides, the R package “REMODBio” has been published in the EMODnet Github. This package allows users to download data using the EMODnet Biology Web Feature Service (WFS). The user can download the data applying multiple filters, e.g.: using the species unique identifier (aphiaID), the dataset unique identifier (dasID), or by providing a geoserver URL that can be obtained through the EMODnet Biology download toolbox.

Figure: Screen capture of the REDMOBio package in Github, a package published by EMODnet Biology to download data using the WFS services.

EMODnet Biology has published support tools for data providers. In October 2018, an online course was launched that gives an introduction to the EMODnet Biology data format and that guides you through all the necessary processing steps, from dataset describing to publishing and quality checking. The course is developed in the Ocean Teacher platform and can be accessed by anyone by self-enrolment. Additionally, an R-Shiny based online QC tool or biological datasets published through IPT was launched to help data providers to evaluate the quality of the data and make the necessary amendments before the data is harvested in EMODnet Biology. EMODnet Biology is closely working with the SeaDataCloud project to develop a transformation and QC tool to apply the same quality standards to the biological data published in SDN.

Figure: Screen capture of an example of the feedback given by the EMODnet Biology onlince QC tool.

Finally, the EMODnet Biology contract has been extended for two more years. In May 2019, the consortium met in Lisbon to discuss the progress made during the first two years and to start advancing on the work plan of the coming two years. A significant effort will focus on improving the data workflows to automatize the update of datasets and increase the FAIRness of the data.

Update on EMODnet Data Ingestion

The ‘EMODnet Ingestion and safe-keeping of marine data’ project seeks to identify and reach out to organisations from research, public, and private sectors who are holding marine datasets and who are not yet connected and contributing to the existing marine data management infrastructures which are driving EMODnet. Those potential data providers should be motivated and supported to release their datasets for safekeeping and subsequent freely distribution and publication through EMODnet. The project started 19th May 2016 and is undertaken by a European consortium of 44 organisations from 29 coastal countries. Most partners are established data centres and the consortium also includes coordinators of the EMODnet thematic data portal projects.

The EMODnet Data Ingestion portal has been launched early February 2017. It encourages data providers to share marine data and provides a submission service and marine data management guidance information. It aims at organisations not yet routinely submitting data sets to national data centres and not yet used to marine data management practices and standards.

Figure: Homepage of EMODnet Data Ingestion

A low threshold is offered by splitting the completion of the submission form in 2 parts, whereby a data submitter only completes a part of the metadata together with the uploading of a data package. Each data submission is assigned to a competent data centre for completing the metadata of the submission. Those complete submissions are then published with their data packages ‘as is’ at the portal in the View Submissions service, where users can search, browse and download the data packages. As a next step assigned data centres elaborate selected submissions further to make (subsets of) the data fit for population into national, European and EMODnet thematic portals. This includes activities like review, validation, conversions to local formats, and further population to the relevant European infrastructures such as SeaDataNet, EurOBIS, and others depending of the theme which feed into EMODnet data portals.

Figure: Data submission workflow

Figure: Homepage of Data Submission service

EMODnet Ingestion is provided with a network of qualified data centres that are divided over many European countries and expert in EMODnet data themes. Currently this network comprises 49 data centres, recruited from the EMODnet Ingestion project consortium and the EMODnet thematic networks.

Figure: Network for elaborating data submissions

The first project year was dedicated to formulating formats and procedures and developing the web portal with its technical services. From the second year onwards most of the efforts was dedicated to marketing and outreach activities in order to promote the EMODnet Ingestion initiative and to gain momentum with the submission service and its processing chain of pathways. For that purpose, promotional materials have been created such as leaflets, bookmarks, posters, stickers, roll-up infographic, standard presentations, and an excellent animation. EMODnet Ingestion has been presented at many European and international occasions, in particular by EMODnet Thematic coordinators, EMODnet Secretariate, EuroGOOS, and at meetings of relevant EU projects like SeaDataCloud, AtlantOS, EuroARGO, ODIP II, CMEMS, and promotional texts and web links have been integrated in each of the EMODnet portals. This has been complemented with national promotion and outreach activities by each of the EMODnet Ingestion consortium members which act as EMODnet ambassadors. They have identified potential sources and actively approached and motivated potential data holders. Also, they gave them active support for making data submissions. And they have organised national meetings in order to make more organisations in their countries aware of EMODnet and ready for EMODnet Ingestion.

This campaign has resulted in a substantial number of portal visitors (monthly average of 8.000 – 9.000 unique visitors), more than 2000 views of the Ingestion animation, and a increasing population of the Submission service and the View Submissions service. At 1st September 2017 there were only 11 submissions and none published. At the 1st July 2019, there are already 676 submissions of which 537 published ‘as is’ and of these 207 also ingested into European portals.

Figure: Division of published submissions over themes

Figure: Data submissions covering cruises of Marine Scotland Science

Figure: Marine Scotland Science data submissions elaborated into CDI entries for wider distribution and use

EMODnet's Open Sea Lab II sets sail!

Are you a Coder? Communicator? Data enthusiast? Or Entrepreneur? Join us at the second EMODnet Open Sea Lab hackathon and develop innovative solutions with marine open data to address blue society challenges! During three days, in the vibrant and historical city of Ghent (Belgium), teams will compete and bring their expertise to develop novel marine and maritime applications using EMODnet, ICES and Copernicus Marine’s wealth of marine data and services.

No previous experience is required! Experts in European marine data management and digital innovation will help build your team (if you don't already have one), lead workshops and coach you to bring your innovative ideas into working solutions.

Challenges to address are to be found in three main categories:

• Sustainable Blue Economy
• Blue Society & Ocean literacy
• Marine environment protection & management (including climate change);

The OSL II Hackathon takes place 4-6 September 2019 in Ghent – Belgium. There will be around 20 coaches & experts on-site from EMODnet, ICES, Copernicus Marine & Imec. Many workshops available, from Co-creation and Business modelling to Machine learning, Data visualization and Story mapping. Tools for participants, prizes, side events, inspirational speakers and many networking opportunities!

Register at the OSL II website for participating!

SeaDataNet and the Pilot Blue Cloud project

The European Open Science Cloud (EOSC) is an initiative launched by the European Commission in 2016, as part of the European Cloud Initiative. The objective of the EOSC is to provide a virtual environment with free at the point of use, open and seamless services for storage, management, analysis and re-use of research data, across borders and scientific disciplines, leveraging and federating the existing data infrastructures. In short, the aim of EOSC is to create a global, federated environment for managing and sharing scientific data, for science.

Following this launch in the meantime several Calls have been published and several projects have been granted for developing (parts of) the EOSC, such as for example ENVRIFAIR. In the marine domain a Flagship Call was published for ‘The Future of Seas and Oceans Flagship Initiative’. This can be considered as focusing on a thematic EOSC cloud while also being very relevant for the Blue Economy, Marine Environment and Marine Knowledge agendas.

Responding to the Call a proposal for a ‘Pilot Blue Cloud’, dedicated to marine data management, was successfully submitted. It is coordinated by Trust-IT with MARIS as technical coordinator. The aims of the project are:

• To explore and demonstrate the potential of cloud based open science
• To build and demonstrate a Pilot Blue Cloud by combining distributed marine data resources, computing platforms, and analytical services
• To develop services for supporting research to better understand & manage the many aspects of ocean sustainability
• To formulate a roadmap for a longer-term expansion of the Blue Cloud infrastructure and services, e.g. as a thematic EOSC cloud

The Blue-Cloud project will federate leading European marine data management infrastructures (SeaDataNet, EurOBIS, Euro-Argo, Argo GDAC, EMODnet, ELIXIR-ENA, EuroBioImaging, CMEMS, C3S, and ICOS-Marine), and horizontal e-infrastructures (EUDAT, DIAS, D4Science) to capitalise on what exists already and to develop and deploy the Blue Cloud. The federation will be at the levels of data resources, computing resources and analytical service resources. A Blue Cloud data discovery and access service will be developed to facilitate sharing with users of multi-disciplinary datasets. A Blue Cloud Virtual Research Environment (VRE) will be established to facilitate that computing and analytical services can be shared and combined for specific applications.

This innovation potential will be explored and unlocked by developing five dedicated Demonstrators together with excellent marine researchers. This will include configuring on the VRE a number of Virtual Labs, one for each Demonstrator. Each Virtual Lab will be configured as a combination of analytical and generic services, selected from the overall service portfolio and as provided by the computing platforms. There is already a large portfolio of existing services managed by the Blue Cloud founders. These will be used for developing and configuring the VRE and the Virtual Labs, whereby upgrading of selected existing services and developing additional services might take place for providing the planned Demonstrators their full requested functionality.

The Demonstrators will be instrumental for validating the technical concepts and will also be used to promote the potential of the Blue Cloud towards researchers by means of workshops and policy and decision makers by means of demonstration events as their feedback will provide input for a Blue-Cloud roadmap for a longer-term expansion and evolution of the Blue Cloud.

The project will start in November 2019 and has a 3 years duration.

Acronyms as used in this Newsletter

This newsletter contains many acronyms which are described in the following list: