Difference between revisions of "Data Formats and Submission for ACs and UFRs"

From KBwiki
Jump to navigation Jump to search
(New page: QNET-CFD KNOWLEDGE BASE The Knowledge Base for Quality and Trust in the Industrial Application of Computational Fluid Dynamics QNET-CFD Knowledge Base Data Formats and Submissio...)
 
Line 1: Line 1:
QNET-CFD
+
= QNET-CFD Knowledge Base Data Formats and Submission =
 
 
KNOWLEDGE BASE
 
 
 
The Knowledge Base for Quality and Trust
 
 
 
in the Industrial Application of Computational Fluid Dynamics
 
 
 
 
 
 
 
 
 
QNET-CFD Knowledge Base
 
Data Formats and Submission
 
 
   
 
   
                                                                                     
+
 
  
13th November 2001
+
==Purpose of Document ==
 
 
 
 
 
 
 
 
1.    Purpose of Document
 
  
 
This document provides guidance to members who are submitting data to be included in the QNET-CFD Knowledge Base. It is important that members read the content carefully before preparing and submitting their data files. This document sets out the required format for each type of data file, what metadata is required and the folder structures that should be used when submitting data. The total storage space available per Application Challenge will be subject to a limit of 500 MB.
 
This document provides guidance to members who are submitting data to be included in the QNET-CFD Knowledge Base. It is important that members read the content carefully before preparing and submitting their data files. This document sets out the required format for each type of data file, what metadata is required and the folder structures that should be used when submitting data. The total storage space available per Application Challenge will be subject to a limit of 500 MB.
Line 32: Line 13:
 
   
 
   
  
2.    Data Structure and Format
+
==Data Structure and Format==
  
 
   
 
   
  
2.1  Directory Structure
+
===Directory Structure===
  
 
   
 
   
Line 46: Line 27:
 
   
 
   
  
2.2  File Type Segregation
+
===File Type Segregation===
  
 
   
 
   
Line 54: Line 35:
 
   
 
   
  
Graphical Data Files (independent of size)
+
* Graphical Data Files (independent of size)
  
 
   
 
   
  
Small Data Files - Experimental or CFD data files that are less than about 2Mb when stored in ASCII format.  
+
* Small Data Files - Experimental or CFD data files that are less than about 2Mb when stored in ASCII format.  
  
 
   
 
   
  
Large Data Files - Experimental or CFD data files that would be more than about 2Mb if stored in ASCII format.
+
* Large Data Files - Experimental or CFD data files that would be more than about 2Mb if stored in ASCII format.
  
 
   
 
   
Line 72: Line 53:
 
   
 
   
  
2.3  Graphical Data Files (independent of size)
+
=== Graphical Data Files (independent of size)===
  
 
   
 
   
Line 86: Line 67:
 
   
 
   
  
2.4  Small Data files
+
=== Small Data files ===
  
 
   
 
   
Line 96: Line 77:
 
   
 
   
  
2.5  Large Data files
+
=== Large Data files ===
  
+
‘Large’ data files are experimental or CFD data that are larger than 2Mb, if stored in ASCII format. The preferred format for large data files is the HHDB (Houston Hypersonic Database) format, details of which can be obtain from the web site at:
  
‘Large’ data files are experimental or CFD data that are larger than 2Mb, if stored in ASCII format. The preferred format for large data files is the HHDB (Houston Hypersonic Database) format, details of which can be obtain from the web site at
+
* [http://www-sop.inria.fr/sinus/Softs/Vigie/HHDB/hhdb.html] 
  
http://www-sop.inria.fr/sinus/Softs/Vigie/HHDB/hhdb.html .  All HHDB files must have the .hhdb extension
+
All HHDB files must have the .hhdb extension
  
 
   
 
   
Line 108: Line 89:
 
   
 
   
  
There are conversion routines between ASCII and HHDB formats available on the QNET-CFD Knowledge base site at http://eddie.mech.surrey.ac.uk/Help/utils/
+
There are conversion routines between ASCII and HHDB formats available on the QNET-CFD Knowledge base site at [http://eddie.mech.surrey.ac.uk/Help/utils/]
  
 
   
 
   
Line 116: Line 97:
 
   
 
   
  
1.      CGNS standardised binary format (.cgns extension)
+
* CGNS standardised binary format (.cgns extension)
  
2.      Native format of commercial CFD code (use a suitable extension)
+
* Native format of commercial CFD code (use a suitable extension)
  
 
   
 
   
Line 126: Line 107:
 
   
 
   
  
 
+
== Data Submission, Storage and Retrieval ==
 
 
 
 
 
3.    Data Submission, Storage and Retrieval
 
  
 
   
 
   
Line 136: Line 113:
 
   
 
   
  
3.1  Data Submission Methods
+
=== Data Submission Methods ===
  
 
   
 
   
  
·        FTP (for data sets of the order of tens of MB or less)
+
* FTP (for data sets of the order of tens of MB or less).  '''This is done by special arrangement with the QNET Editorial team.'''
  
·        CD ROM or DLT (for data sets of the order of hundreds of MB)
+
* CD ROM, DVD or DLT (for data sets of the order of hundreds of MB)
  
 
   
 
   
  
Inform the KB curator of ftp access via <p.voke@surrey.ac.uk>.Mail hard media to Prof. Peter R Voke, School of Engineering (H5), University of Surrey, Guildford GU2 7XH, U.K.
+
Inform the KB curator of ftp access via email. Hard media should be sent to .....
 
 
 
   
 
   
  
 
   
 
   
  
3.2 Data Storage
+
=== Data Storage ===
  
 
   
 
   
Line 160: Line 136:
 
   
 
   
  
3.3  Data Retrieval
+
=== Data Retrieval===
  
 
   
 
   
Line 170: Line 146:
 
   
 
   
  
4.    Appendix A: Metadata Required
+
==  Appendix A: Metadata Required ==
  
 
   
 
   
Line 177: Line 153:
  
 
   
 
   
 
+
* Values of all the defining parameters specified for the Application Challenge which do not explicitly appear as a separate column in the data file. These rows should be prefixed by a # symbol.
1. Values of all the defining parameters specified for the Application Challenge which do not explicitly appear as a separate column in the data file. These rows should be prefixed by a # symbol.
 
  
 
   
 
   
  
2. The Thematic Area, using machine-readable format:
+
* The Thematic Area, using machine-readable format:
  
 +
<pre>
 
%TA = <n>
 
%TA = <n>
 +
</pre>
 +
where <n> = 1, 2, 3, 4, 5 or 6.
  
where <n> = 1, 2, 3, 4, 5 or 6.
 
  
 
   
 
   
  
32. The Application Challenge number, using machine readable format:
+
* The Application Challenge number, using machine readable format:
  
 +
<pre>
 
%AC = <nTA-N>
 
%AC = <nTA-N>
 +
</pre>
  
 
where <nTA> = 1 to 45.is the Thematic Area number and N is the AC number, e.g. 4-01 refers to the first AC in TA4 “Wind environment around an airport terminal building” by WS Atkins.
 
where <nTA> = 1 to 45.is the Thematic Area number and N is the AC number, e.g. 4-01 refers to the first AC in TA4 “Wind environment around an airport terminal building” by WS Atkins.
Line 206: Line 185:
 
   
 
   
  
43. The variables listed for each column of data using the machine-readable format, for  
+
* The variables listed for each column of data using the machine-readable format, for  
  
 
example:
 
example:
  
 
   
 
   
 +
<pre>
  
 
%Column1Name = VelocityX
 
%Column1Name = VelocityX
Line 223: Line 203:
  
 
%Column2Type = String  
 
%Column2Type = String  
 +
 +
</pre>
  
 
etc. for each column in the file
 
etc. for each column in the file
Line 228: Line 210:
 
   
 
   
  
The value assigned to %Column1Name must be a string containing no unprintable characters (apart from spaces) and if possible it should conform to the CGNS convention for data name identifiers. For details of CGNS conventions, see http://www.grc.nasa.gov/www/cgns/sids/dataname.html  
+
The value assigned to %Column1Name must be a string containing no unprintable characters (apart from spaces) and if possible it should conform to the CGNS convention for data name identifiers. For details of CGNS conventions, see [http://www.grc.nasa.gov/www/cgns/sids/dataname.html]
  
 
   
 
   
Line 244: Line 226:
 
   
 
   
  
54. The metadata must end with the following machine readable-command:
+
* The metadata must end with the following machine readable-command:
 
+
<pre>
 
%End Of Metadata
 
%End Of Metadata
 
+
</pre>
 
   
 
   
  
Line 255: Line 237:
  
 
   
 
   
 +
== Appendix B: Example of Small data ASCII files ==
  
5.    Appendix B: Example of Small data ASCII files
+
<pre>
 
 
 
############################################################
 
############################################################
  
Line 385: Line 367:
  
  
 +
--------------------------------------------------------------------------------
 +
 +
 +
</pre>
  
--------------------------------------------------------------------------------
 
  
1 The name of the folder should be the Application Challenge  number in the form: number <AC><TA>-<N>, where TA is the Thematic Area number and N is the AC number, e.g. AC4-01 refers to the first AC in TA4 “Wind environment around an airport terminal building” by WS Atkins.<n>, where <n> = 1 to 45. Note that the Application Challenge number corresponds to the QNET-CFD member  
+
The name of the folder should be the Application Challenge  number in the form: number <AC><TA>-<N>, where TA is the Thematic Area number and N is the AC number, e.g. AC4-01 refers to the first AC in TA4 “Wind environment around an airport terminal building” by WS Atkins.<n>, where <n> = 1 to 45. Note that the Application Challenge number corresponds to the QNET-CFD member  
  
 
number (i.e. 41 for University of Surrey, 2 for WS Atkins). If any member is to submit  
 
number (i.e. 41 for University of Surrey, 2 for WS Atkins). If any member is to submit  

Revision as of 12:15, 6 August 2009

QNET-CFD Knowledge Base Data Formats and Submission

Purpose of Document

This document provides guidance to members who are submitting data to be included in the QNET-CFD Knowledge Base. It is important that members read the content carefully before preparing and submitting their data files. This document sets out the required format for each type of data file, what metadata is required and the folder structures that should be used when submitting data. The total storage space available per Application Challenge will be subject to a limit of 500 MB.




Data Structure and Format

Directory Structure

All the data files associated with an Application Challenge should be submitted as a single appropriately named folder[1], containing three subfolders named ‘X’, ‘C’ and ‘I’. All experimental data must be contained within the directory named ‘X’, all CFD data in the directory named ‘C’ and all images in a directory named ‘I’. Any further directory structure within these three subfolders and the data filenames, are at the discretion of the Member providing the data. The total storage space available per Application Challenge will be subject to a limit of 500 MB.



File Type Segregation

Data stored in the Knowledge Base are separated into the following three classes:


  • Graphical Data Files (independent of size)


  • Small Data Files - Experimental or CFD data files that are less than about 2Mb when stored in ASCII format.


  • Large Data Files - Experimental or CFD data files that would be more than about 2Mb if stored in ASCII format.


In the following sections the file formats required for each of these three classes will be discussed and must be adhered to when submitting documents.



Graphical Data Files (independent of size)

Static images must have extensions of either “.gif” or “.jpg” or “.png” depending on which format is the most suitable for a given image type.


Files containing movie clips should have extensions of either “.mpg” or “.avi” or “.mov”.



Small Data files

‘Small’ data files are experimental or CFD data files that are smaller than about 2Mb when stored in ASCII format. These These data files must be in ASCII format, and should have the extension .dat. They and must should contain their own metadata at the top of the file as indicated in Appendix A;. tThis metadata header states the Problem Definition Parameters (PDPs) common to the entire data file and also information about the data columns presented. After the metadata header, the file should only contain columns of tab separated data with the rows being separated by CR/LF (Carriage Return/Line Feed). The data type in each column must be either Integer, Real or String, as declared in the metadata. No metadata or any other type of comments should appear after the “%End of Metadata” line has been encountered. An example of a small data file is provided in Appendix B.



Large Data files

‘Large’ data files are experimental or CFD data that are larger than 2Mb, if stored in ASCII format. The preferred format for large data files is the HHDB (Houston Hypersonic Database) format, details of which can be obtain from the web site at:

All HHDB files must have the .hhdb extension



There are conversion routines between ASCII and HHDB formats available on the QNET-CFD Knowledge base site at [2]


If the use of other formats is unavoidable, the following order of preference should be followed:


  • CGNS standardised binary format (.cgns extension)
  • Native format of commercial CFD code (use a suitable extension)


The metadata associated with a large data file must be stored in a separate ASCII file residing in the same directory as the large data file and having the same name except with a “.met” extension. This metadata file shall contain all the Problem Definition Parameters (PDPs) common to the entire data file and also information about the data columns presented as prescribed in Appendix A.


Data Submission, Storage and Retrieval

Data Submission Methods

  • FTP (for data sets of the order of tens of MB or less). This is done by special arrangement with the QNET Editorial team.
  • CD ROM, DVD or DLT (for data sets of the order of hundreds of MB)


Inform the KB curator of ftp access via email. Hard media should be sent to .....



Data Storage

The data will be stored in the format in which they were submitted. An exception can be made if ASCII data following the structure of a binary HHDB file is submitted with a request that it be converted.


Data Retrieval

Small data files will be downloadable in the format in which they were submitted.. For large data files it is proposed that a CGI program will be implemented on the Knowledge Base web site enabling users to download just a portion of a large file which is of interest to them. It is envisaged that downloads will be limited to about 10Mb to prevent overloading the web.



Appendix A: Metadata Required

The metadata (data about data) which must be provide for each data file shall contain the following information in ASCII format:


  • Values of all the defining parameters specified for the Application Challenge which do not explicitly appear as a separate column in the data file. These rows should be prefixed by a # symbol.


  • The Thematic Area, using machine-readable format:
%TA = <n>

where <n> = 1, 2, 3, 4, 5 or 6.



  • The Application Challenge number, using machine readable format:
%AC = <nTA-N>

where <nTA> = 1 to 45.is the Thematic Area number and N is the AC number, e.g. 4-01 refers to the first AC in TA4 “Wind environment around an airport terminal building” by WS Atkins.

Note that the Application Challenge number corresponds to the QNET-CFD member

number (i.e. 41 for University of Surrey, 2 for WS Atkins). If any member is to submit

more than one AC, then the convention is to use the member number followed by

a,b,c,….(e.g. 8a, 8b).


  • The variables listed for each column of data using the machine-readable format, for

example:



%Column1Name = VelocityX

%Column1Units = m/s

%Column1Type = Real 

%Column2Name = Sensor Location on Wing

%Column2Units = mm

%Column2Type = String 

etc. for each column in the file


The value assigned to %Column1Name must be a string containing no unprintable characters (apart from spaces) and if possible it should conform to the CGNS convention for data name identifiers. For details of CGNS conventions, see [3]


The value assigned to %Column1Units is left to the discretion of the members but once again it must be a string containing no unprintable characters except spaces.


The value assigned to %Column1Type must be either “Integer”, “Real” or “String”


Entries are case-sensitive and should follow the exact specification given.


  • The metadata must end with the following machine readable-command:
%End Of Metadata




Appendix B: Example of Small data ASCII files

############################################################

#

# TA1 External Aerodynamics AC41 Oscillating Delta Wing

#

# File contains ANOVA-results of the Pressure Coefficients -Cp

# of the Kulite Pressure Transducers

#

# Free stream Velocity : 40 ms-1 (± 0.06 %)

# Reynolds number      : 3.1E6

# Mach number          : 0.12

# Mode                 : Pitch

# Angle of Attack      : 0.0 deg

# Oscillation Amplitude           : 6 deg

############################################################

%TA = 1

%AC = 1-4110

%Column1Name = value for red. Frequ. = 0.28

%Column1Units = rad/s

%Column1Type = Real 

%Column2Name = value for red. Frequ. = 0.56

%Column2Units = rad/s

%Column2Type = Real

%Column3Name = confidence interval for 95% probability

%Column3Units = Non-dimensional

%Column3Type = Real

%Column4Name = Side Of Wing

%Column4Units = Location

%Column4Type = String

%Column5Name = x/ci

%Column5Units = ratio

%Column5Type = Real

%Column6Name = eta

%Column6Units = degrees

%Column6Type = Real

%End Of Metadata

0.0931     0.0907     0.0080     Right      0.3        0.700

0.2934     0.3035     0.0010     Right      0.3        0.700

-4.4748    -9.6125    0.0600     Right      0.3        0.700

0.0798     0.0869     0.0010     Right      0.3        0.700

-17.5862   -37.9607   0.3240     Right      0.3        0.700

0.0230     0.0274     0.0000     Right      0.3        0.700

-25.8954   -50.8786   1.6030     Right      0.3        0.700

0.1060     0.1104     0.0020     Left       0.3        0.700

0.2874     0.2975     0.0010     Left       0.3        0.700

-4.5508    -9.6137    0.1160     Left       0.3        0.700

0.0779     0.0848     0.0010     Left       0.3        0.700

-18.4120   -39.0618   0.6240     Left       0.3        0.700

0.0219     0.0265    0.0010     Left       0.3        0.700

-28.0049   -54.2960   2.2470     Left       0.3        0.700

0.1006     0.1015    0.0020     Left       0.3        0.850

0.3287     0.3403     0.0010     Left       0.3        0.850

-1.4226    -3.4499    0.0370     Left       0.3        0.850

0.0818     0.0866     0.0010     Left       0.3        0.850

-0.8961    -4.9886    0.3980     Left       0.3        0.850

0.0118     0.0166     0.0000     Left       0.3        0.850

22.3086    32.6002    2.5760     Left       0.3        0.850

0.0992     0.0834     0.0050     Left       0.6        0.700

0.1712     0.1730     0.0010     Left       0.6        0.700

0.4754     1.7458     0.1840     Left       0.6        0.700

0.0437     0.0487     0.0010     Left       0.6        0.700

-34.5128   -66.4447   1.0530     Left       0.6        0.700

0.0400     0.0444     0.0010     Left       0.6        0.700

-43.8689   -84.7527   1.1160     Left       0.6        0.700


--------------------------------------------------------------------------------



The name of the folder should be the Application Challenge number in the form: number <AC><TA>-<N>, where TA is the Thematic Area number and N is the AC number, e.g. AC4-01 refers to the first AC in TA4 “Wind environment around an airport terminal building” by WS Atkins.<n>, where <n> = 1 to 45. Note that the Application Challenge number corresponds to the QNET-CFD member

number (i.e. 41 for University of Surrey, 2 for WS Atkins). If any member is to submit

more than one AC, then the convention is to use the member number followed by

a,b,c,….(e.g. 8a, 8b).