Data storage guidelines for DNS cases: Difference between revisions
No edit summary |
|||
| (2 intermediate revisions by one other user not shown) | |||
| Line 1: | Line 1: | ||
= HDF5 files = | |||
There are numerous file types and formats for storing CFD data. One popular format for large datasets is [https://www.hdfgroup.org/solutions/hdf5/ HDF5] which has excellent cross-platform support and fast I/O. We recommend HDF5 as an underlying file format for DNS and other large dataset contributions. | |||
= Storing high fidelity CFD data = | = Storing high fidelity CFD data = | ||
High fidelity CFD data is generally too large to be stored directly on the wiki and so a special cloud based storage service has been created. It is based on the AWS S3 service and contributors can request access to upload data to it when creating a new DNS/LES case in the wiki. To do this contact the KB Wiki Team team (contact via admin@cado-ercoftac.org). For more information on the templates for high fidelity data see [[Article Templates]]. | High fidelity CFD data is generally too large to be stored directly on the wiki and so a special cloud based storage service has been created. It is based on the AWS S3 service and contributors can request access to upload data to it when creating a new DNS/LES case in the wiki. To do this contact the KB Wiki Team team (contact via admin@cado-ercoftac.org). For more information on the templates for high fidelity data see [[Article Templates]]. | ||
| Line 30: | Line 33: | ||
== Add links to you article == | == Add links to you article == | ||
Once your data has uploaded you can create links within your article to the S3 data. These will be in the form https://kbwiki-data.s3.eu-west-2.amazonaws.com/DNS-23-example/my_data_file.example | Once your data has uploaded you can create links within your article to the S3 data. These will be in the form https://kbwiki-data.s3.eu-west-2.amazonaws.com/DNS-23-example/my_data_file.example | ||
You can also browse the S3 data to verify the upload and copy the link URLs at the following URL: | |||
http://kbwiki-data.s3-eu-west-2.amazonaws.com/index.html | |||
'''Please note''' that the high fidelity CFD data may be extremely large and so we would encourage that you include the file size next to any links to avoid users downloading large datasets when not required. | '''Please note''' that the high fidelity CFD data may be extremely large and so we would encourage that you include the file size next to any links to avoid users downloading large datasets when not required. | ||
== Assistance == | |||
For assistance with large data files and accessing the DNS test cases produced under the HiFi-Turb project, contact [mailto:admin@cado-ercoftac.org?Subject=ERCOFTAC%20KB%20Wiki%20DNS%20Cases&body=Text KB Wiki Team - DNS Data] | |||
Latest revision as of 08:42, 4 May 2023
HDF5 files
There are numerous file types and formats for storing CFD data. One popular format for large datasets is HDF5 which has excellent cross-platform support and fast I/O. We recommend HDF5 as an underlying file format for DNS and other large dataset contributions.
Storing high fidelity CFD data
High fidelity CFD data is generally too large to be stored directly on the wiki and so a special cloud based storage service has been created. It is based on the AWS S3 service and contributors can request access to upload data to it when creating a new DNS/LES case in the wiki. To do this contact the KB Wiki Team team (contact via admin@cado-ercoftac.org). For more information on the templates for high fidelity data see Article Templates.
How to upload the data
The KB Wiki Team team will supply you with an account key, a secret key and a path. Once you have these you can use any S3 compatible tool to upload your data. The instructions below are for the AWS CLI tool.
Install AWS CLI
The installation instructions for AWS CLI can be found here.
Configure AWS CLI
Next configure AWS CLI with the details supplied by the KB Wiki Team. For example:
>aws configure
AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-west-2
Default output format [None]: json
Upload data
AWS CLI provides several commands to list, upload and download data. For copying an entire directory to the wiki data store we would recommend using the s3 sync command. The KB Wiki Team will provide a path for you to upload to, in this example we are uploading from a local "~/data" directory to the path "DNS-23-example" provided by the KB Wiki Team. We do a dry-run test first to check it is uploading the data we expect before we do the actual copy.
> cd ~/data
> aws s3 sync . s3://kbwiki-data/DNS-23-example/ --dryrun
> aws s3 sync . s3://kbwiki-data/DNS-23-example/
Add links to you article
Once your data has uploaded you can create links within your article to the S3 data. These will be in the form https://kbwiki-data.s3.eu-west-2.amazonaws.com/DNS-23-example/my_data_file.example
You can also browse the S3 data to verify the upload and copy the link URLs at the following URL:
http://kbwiki-data.s3-eu-west-2.amazonaws.com/index.html
Please note that the high fidelity CFD data may be extremely large and so we would encourage that you include the file size next to any links to avoid users downloading large datasets when not required.
Assistance
For assistance with large data files and accessing the DNS test cases produced under the HiFi-Turb project, contact KB Wiki Team - DNS Data