Description

Contains all solar wind data coming from observation or modelisation.
Fetch files from some providers (observatories, models), download them locally in DATA_OBS and DATA_MODEL tree, and add some metadata to a SQLite database in DATA_DB folder.
Files are not stored in the database because they are too large (up to some Mb).
There are 2 ways to access to these files:

manually dive into the tree and find the good file
ask the database for a file with a given type and date, the database will return the path of a corresponding file.

Installation

See Readme page.

Data type

Basic data types:

common.module_timespace.Time: represent a time in different format (UTC, Python datetime, Ephemeride Time).
common.module_timespace.Coordinate: represent a position in a 3D space (polar and cartesian coordinates).

Observational data types:

common.module_chole.Chole: represent a Coronal Hole. Contains npoint module_timespace.Coordinate() objects following the Coronal Hole contour.
common.module_cube3D.Cube3D: represent a magnetic datacube. Contains 3 matrix br,blon,blat of 3D cartesian dimensions.
common.module_event.Event: represent an event with its type (CME,...), begin or peak or ending date, location.
common.module_fieldline.Fieldline: represent a magnetic fieldline. Contains 1D-vector of br,blon,blat and its coordinates.
common.module_map2D.Map2D: represent a scalar field on a 2D-surface (radial magnetic field, light intensity).
common.module_timeserie.Timeserie: represent a timeserie of any data. Data is a 1D-vector of n-tuples variyin with time.

Fetching the data

Which files are fetched?
Look at remote servers for new files (not downloaded yet) or files which are newer than the corresponding file in the database.
Fetch levels and data formats.
Files are integrated in the database following different fetch levels and data formats.

Fetch level: the original downloaded file has a fetch level = 0. Then some operations may be done to create other files from it (gunzip, reformat data, create a symbolic link with a more friendly name), each operation keep the previous file with fetch level n and create a new file with a fetch level n+1.
Data format: file with different fetch levels can have different data formats or can have the same data format (if only the name has changed for instance).

Here the list of Map2D fetch type and data format:

Fetch level and data format for Map2D
Table name	Fetch level 0	Data format	Fetch level 1	Data format	Fetch level 2	Data format
OBS_MAPS_NSO	mrmqs190809t0653c2220_000.fits.gz	Gzip	mrmqs190809t0653c2220_000.fits	FITS_NSO_V0	nso_gong_CR2220.fits	FITS_NSO_V0
OBS_MAPS_NSOZ	mrzqs200704t1804c2232_079.fits.gz	Gzip	mrzqs200704t1804c2232_079.fits	FITS_NSO_V0	nso_gongz_20200704180400.fits	FITS_NSO_V0
OBS_MAPS_ADAPT40	adapt40311_03k012_202007190600_f03002700n0.fts.gz	Gzip	adapt40311_03k012_202007190600_f03002700n0.fts	FITS_ADAPT_V0	adapt41_gong_20200719060000.fits	FITS_ADAPT_V0
OBS_MAPS_ADAPT41	adapt41311_03k012_202007190600_f03002700n0.fts.gz	Gzip	adapt41311_03k012_202007190600_f03002700n0.fts	FITS_ADAPT_V0	adapt41_gong_20200719060000.fits	FITS_ADAPT_V0
OBS_MAPS_WSO_ORIGIN	WSO.2206.txt	ASCII_WSO_V0	V1_WSO.2206.txt	ASCII_WSO_V1	wso_origin_CR2206.txt	ASCII_WSO_V1
OBS_MAPS_WSO_FILLED	WSO.2206.F.txt	ASCII_WSO_V0	V1_WSO.2206.F.txt	ASCII_WSO_V1	wso_filled_CR2206.txt	ASCII_WSO_V1
OBS_MAPS_WSO_DATE	Photo.latest.txt	ASCII_WSO_V0	Photo_latest_20191127214327.txt	ASCII_WSO_V0	wso_latest_20191127214327.tx	ASCII_WSO_V1
OBS_MAPS_SDO_AIA_XXX	CR2186.fits	FITS_SDOAIA_V0	sdo_aia171_CR2186.fits	FITS_SDOAIA_V0	Null	Null
OBS_MAPS_SOHO_EIT_XXX	CR2053_171A_c.fits	FITS_SOHOEIT_V0	soho_eit171-1_CR2053.fits	FITS_SOHOEIT_V0	Null	Null
OBS_MAPS_SOHO_EIT_XXX	171_cr2058.fits	FITS_SOHOEIT_V0	soho_eit171-2_CR2058.fits	FITS_SOHOEIT_V0	Null	Null
OBS_MAPS_EUV2CARMAP_SDOSTA_XXX	2020Jul23T1200CRMAP_EUV.fits	FITS_EUV2CARMAPSDOSTA_V0	euv2carmap_sdosta171_20200723120000.fits	FITS_EUV2CARMAPSDOSTA_V0	Null	Null
OBS_MAPS_EUV2CARMAP_SDO360_XXX	CRMAP_SDO360_AIA171_20200723T1800.fits	FITS_EUV2CARMAPSDO360_V0	euv2carmap_sdo360171_20200723180000.fits	FITS_EUV2CARMAPSDO360_V0	Null	Null
OBS_MAPS_WL2CARMAP_SOHOLASCO_C2	WL_CRMAP_20200720T120000_LC2_2p5Rs_merged.fits	FITS_WL2CARMAPSOHOLASCOC2_V0	wl2carmap_soholascoc2_20200720120000.fits	FITS_WL2CARMAPSOHOLASCOC2_V0	Null	Null

Here the list of Timeserie fetch type and data format:

Fetch level and data format for Timeserie
Table name	Fetch level 0	Data format	Fetch level 1	Data format	Fetch level 2	Data format
OBS_INSITU_MAG_ACELV4	ACE_MAG16_2018-311_V3-3.zip	Zip	ACE_MAG16_2018-311_V3-3	ASCII_ACEMAGLV4_V0	ace_maglv4_20181107000000.ascii	ASCII_ACEMAGLV4_V1
OBS_INSITU_VELOCITY_ACELV4	20180501_ace_swepam_1m.txt	ASCII_ACEVELOCITYLV4_V0	ace_velocitylv4_20180501000000.txt	ASCII_ACEVELOCITYLV4_V0	Null	Null
OBS_INSITU_MAG_DISCOVER	oe_m1m_dscovr_s20181031000000_e20181031235959_p20181101031426_pub.nc.gz	Gzip	oe_m1m_dscovr_s20181031000000_e20181031235959_p20181101031426_pub.nc	NETCDF_DISCOVERMAG_V0	discover_mag_20181031000000.nc	NETCDF_DISCOVERMAG_V1
OBS_INSITU_VELOCITY_DISCOVER	oe_f1m_dscovr_s20180501000000_e20180501235959_p20180502034038_pub.nc.gz	Gzip	oe_f1m_dscovr_s20180501000000_e20180501235959_p20180502034038_pub.nc	NETCDF_DISCOVERVELOCITY_V0	discover_velocity_20180501000000.nc	NETCDF_DISCOVERVELOCITY_V1
OBS_INSITU_MAG_STEREOA	STA_LB_IMPACT_20181031_V02.cdf	CDF_STEREOIMPACT_V0	stereoa_mag_20181031000000.cdf	CDF_STEREOIMPACT_V1	Null	Null
OBS_INSITU_VELOCITY_STEREOA	STA_LB_PLASTIC_20181031_V12.cdf	CDF_STEREOPLASTIC_V0	stereoa_velocity_20181031000000.cdf	CDF_STEREOPLASTIC_V1	Null	Null

Here the list of Event fetch type and data format:

Fetch level and data format for Timeserie
Table name	Fetch level 0	Data format	Fetch level 1	Data format	Fetch level 2	Data format
OBS_CORONA_SOLARMONITOR_NOAAEVENT	noaa_events_raw_20200726.txt	ASCII_SOLARMONITORNOAAEVENT_V0	solarmonitor_noaaevent_20200726000000.txt	ASCII_SOLARMONITORNOAAEVENT_V0	Null	Null
OBS_CORONA_DONKI_CME	2020-07-25T01:25:00-CME-001	JSON_DONKICME_V0	donki_event_20200726131200.txt	JSON_DONKICME_V0	Null	Null

Here the list of Coronal Hole fetch type and data format:

Fetch level and data format for Coronal Hole
Table name	Fetch level 0	Data format	Fetch level 1	Data format	Fetch level 2	Data format
OBS_CORONA_SOLARMONITOR_CHIMERA	arm_ch_location_20200725.txt	ASCII_SOLARMONITORCHIMERA_V0	solarmonitor_chimera_hgc_20200725185440.txt	ASCII_SOLARMONITORCHIMERA_V0	solarmonitor_chimera_hpc_20200725185440.txt	ASCII_SOLARMONITORCHIMERA_V1

DATA_OBS/Maps

Data at solar surface.

2D Map

Provider: 2D maps come from different observatories, models. Here is the list.

Database: database_map2D script is in charge of all 2D maps of Map section.
Each provider can feed one or more SWD's table.
For rebuiling from scratch the map2D database, see build_database_map2D script.

Fetch

Files are locally download using some fetch information contained in interface_fetch_map2D.
Meta data is stored in the SWD following these attributes.
A shell script can launch all fetch scripts for 2D map: dir.cron/update_database_map2D.
Here the list of fetch scripts:

Read

An interface routine interface_read_map2D.read read a 2D maps for different type.

read_wso_wso_map2D : for WSO table.
read_adapt_gong_map2D : for ADAPT40 and ADAPT41 tables.
read_nso_gong_map2D : for NSO table.
read_sdo_aia_map2D : for SDO_AIA_171 and SDO_AIA_193 tables.
read_sdo_hmi_map2D : for SDO_HMI table.
read_soho_eit_map2D : for SOHO_EIT_171 and SOHO_EIT_193 tables.
read_euv2carmap_sdosta_map2D: for EUV2CARMAP_SDOSTA_171, EUV2CARMAP_SDOSTA_193, EUV2CARMAP_SDOSTA_304 and EUV2CARMAP_SDOSTA_WL tables.

Test

Test:

test_map2D_time2map
test_map2D_read

Unit test:

DATA_OBS/Corona

Data in the corona.

Coronal Hole (Chole)

Provider: Coronal hole come from different observatories. Here is the list.

Database: database_chole script is in charge of all Coronal hole.
Each provider can feed one or more SWD's table.
For rebuiling from scratch the chole database, see build_database_chole script.

Fetch

Files are locally download using some fetch information contained in interface_fetch_chole.
Meta data is stored in the SWD following these attributes.
A shell script can launch all fetch scripts for coronal hole: dir.cron/update_database_chole.
Here the list of fetch scripts:

fetch_solarmonitor_chimera_chole

Read

An interface routine interface_read_chole.read read a coronal hole for different type.

read_chole_solarmonitor_chimera : for SOLARMONITOR_CHIMERA table.

Event

Provider: Event come from different observatories. Here is the list.

Database: database_event script is in charge of all Event.
Each provider can feed one or more SWD's table.
For rebuiling from scratch the event database, see build_database_event script.

Fetch

Files are locally download using some fetch information contained in interface_fetch_event.
Meta data is stored in the SWD following these attributes.
A shell script can launch all fetch scripts for event: dir.cron/update_database_event.
Here the list of fetch scripts:

fetch_donki_cme.py Fetch the file containing CME event.
fetch_donki_file.py Fetch the content of each Donki file.
fetch_solarmonitor_file.py Fetch the file containing Flare and Xflare events.
fetch_solarmonitor_flare.py Fetch the content of each Solarmonitor file.
fetch_solarmonitor_xflare.py Fetch the content of each Solarmonitor file.

Read

Different routines can read different type of event.

read_donki_event.py : for DONKI_FILE table.
read_solarmonitor_event.py : for SOLARMONITOR_FILE table.

Datacube

Empty section.

Fieldline

Empty section.

DATA_OBS/Heliosphere

Data in the heliosphere.

Datacube

Empty section.

DATA_OBS/Insitu

Insitu data.

Timeserie

Provider: Timeserie come from different observatories. Here is the list.

Database: database_timeserie script is in charge of all Timeserie.
Each provider can feed one or more SWD's table.
For rebuiling from scratch the timeserie database, see build_database_timeserie script.

Fetch

Files are locally download using some fetch information contained in interface_fetch_timeserie.
Meta data is stored in the SWD following these attributes.
A shell script can launch all fetch scripts for coronal hole: dir.cron/update_database_timeserie.
Here the list of fetch scripts:

Read

An interface routine interface_read_timeserie.read read a coronal hole for different type.

Chronological table (Crontab)

Time reference is UTC.

00h01 (AM/PM): Update SWD/DATA_OBS/MAPS. Script dir.cron/update_database_map2D calls all fetch scripts listed above. Duration: 3 min if run 2 times a day.
Todo:
Slurm scheduling

Winter bug: Cron script must be independant of local time (winter/summer zone)