Meta Integration® Model Bridge (MIMB)
"Metadata Integration" Solution

MIMB Bridge Documentation

MIMB Import Bridge from Amazon Web Services (AWS) S3 Storage

Bridge Specifications

Vendor Amazon
Tool Name AWS Simple Storage Service (S3)
Tool Version 1.0
Tool Web Site http://aws.amazon.com/s3/
Supported Methodology [Database] Multi-Model via Java API

Import tool: Amazon AWS Simple Storage Service (S3) 1.0 (http://aws.amazon.com/s3/)
Import interface: [Database] Multi-Model via Java API from Amazon Web Services (AWS) S3 Storage
Import bridge: 'AmazonS3' 10.0.0

IMPORTING FROM Amazon Simple Storage Service.

This bridge establishes a connection with a choosed bucket in order to extract the physical metadata. It is critical that the parameters are filled correctly in order to satisfy the local connection requirements on the client workstation that runs the bridge.
Bridge supports following compressed formats:
ZIP (as a compression format, not as archive format)
BZIP
GZIP
LZ4
Snappy (as standard Snappy format, not as Hadoop native Snappy format)

Please refer to the individual parameter's tool tips for more detailed examples.


Bridge Parameters

Parameter Name Description Type Values Default Scope
Access Key Your access key id to sign programmatic requests to AWS services. STRING      
Secret Key Your secret key to sign programmatic requests to AWS services. PASSWORD      
Root directory Set directory containing metadata files or specify it using browsing tool. Bridge provides up to 3 level browsing depth. Don't forget to specify 'Region' parameter for using browsing tool.

Specify * symbol to import from all available buckets under specified region

Bridge uses only s3a protocol to load files.
f.e. s3a://bucket/dir1/dir2
REPOSITORY_SUBSET     Mandatory
Include filter Relative to the root path, case sensitive filter of files expressed using the extended unix glob expression pattern syntax (e.g. '/*.csv' - import files that end with .csv in root folder; **.csv - import all .csv files in any directory level).
STRING      
Exclude filter Similar to the include filter parameter STRING      
Sample size Enter the directory containing metadata files or specify it using browsing tool. Bridge provides up to 3 level browsing depth. NUMERIC      
Partition directories Files-based partition directories' paths.
The bridge tries to detect partitions automatically. It can take a long time when partitions have a lot of files.
You can shortcut the detection process for a partition by specifying it in this parameter.
Specify the partition directory path relative to the Root directory.
Use . to specify the root directory as the partitioned directory.

Separate multiple paths with the , or ; characters.
For example: dir1/dir2 dir3/dir4 dir5
STRING      
Miscellaneous Specify miscellaneous options identified with a -letter and value.

For example, -m 4G -f 100 -j -Dname=value -Xms1G

-m the maximum Java memory size whole number (e.g. -m 4G or -m 2500M ).
-s import system objects that are skipped by default (e.g. -s)
-v set environment variable(s) (e.g. -v var1=value -v var2="value with spaces").
-j the last option that is followed by Java command line options (e.g. -j -Dname=value -Xms1G).
-hadoop key1=val1;key2=val2 to manualy set hadoop configuration options
STRING      

 

Bridge Mapping

Mapping information is not available

Last updated on Tue, 10 Jul 2018 18:37:05

Copyright © Meta Integration Technology, Inc. 1997-2018 All Rights Reserved.

Meta Integration® is a registered trademark of Meta Integration Technology, Inc.
All other trademarks, trade names, service marks, and logos referenced herein belong to their respective companies.