Store - Placeholder class that represents a way of storing data and getting and putting that data, verifying it and so on.
use Store;
# instantiate the store
my $s=Store->new(authmode=>Store::AUTH_PW);
# get storename
$s->name();
# set storename
$s->name("WHATEVER");
# open store to start using it
$s->open();
# get param value for a get parameter
my $val=$s->paramValueGet("PARAMNAME");
# set param value for a get parameter
$s->paramValueGet("PARAMNAME","VALUE");
# get param value for a put parameter
my $val=$s->paramValuePut("PARAMNAME");
# get param value for a del parameter
my $val=$s->paramValueDel("PARAMNAME");
# set param value for a put parameter
$s->paramValuePut("PARAMNAME","VALUE");
# set a param in put-, get- and del parameters
$s->setParam ("PARAMNAME","VALUE);
# get required put-, get and extra-parameters and
# their current value
my %r=$s->paramsRequired();
# put data into store
$s->put();
# get data from store
$s->get();
# delete data from remote store
$s->del();
# get size of remote location
my $size=$s->remoteSize();
# get size of local location
my $size=$s->localSize();
# list local folder/data area
my $list=$s->listLocal("/whatever/path");
# list remote folder/data area
my $list=$s->listRemote("/whatever/path");
# verify the operation on the Store
$s->verify();
# get log of either get- or put-operation
my $log=$s->getLog();
# get put or get exitcode
my $ecode=$s->exitcode();
# get timout value
my $timeout=$s->timeout();
# set timeout value
$s->timeout(10000);
# get wait value
my $wait=$s->wait();
# set wait value
$s->wait(6000);
# check if put, get or del is running
my $running=$s->isRunning();
# check if store is open
my $open=$s->isOpen();
# get success of last get-, put- or del operation
my $succ=$s->success();
# cease get or put operation on a store
$s->ceaseOperations();
# get last error message
my $error=$s->error();
# close the store - clean up. Should be called, even upon error when stopping to use the store.
$s->close();Placeholder class to represent a way of storing data and how to get-, put- and delete data on it, verifying the data, the size, logging and log output of get-, put- and delete operations.
This placeholder class is to be inherited by classes that represent actual ways of storing data.
Constructor. Instantiate a Store-class.
Input accepts the following parameters:
authmode Sets the authentication mode of the Store. It can either be AUTH_PW (password is specifed as input parameters password), AUTH_PWFILE (password is read from file set in parameter passwordfile), AUTH_KEY (uses privatekey specified as input parameter privatekey) or AUTH_KEYFILE (uses privatekey read from file name and location specified as input parameter privatekeyfile). All these parameters (password, passwordfile, privatekey and privatekeyfile) are parameters to the open()-method and are reserved for this (not to be used for other purposes). They might not have meaning for all classes, such as eg. FTP, which do not use a privatekey-scheme for authentication.
metadata Sets the metadata of the dataset that is being put/get/deleted. HASH-reference. Optional. If not set or set to non-HASH reference, will default to an empty HASH. It is up to the inheriting Store-class to use this metadata or not.
storename Sets the unique store name. If none is specified it defaults to a 64 character random string.
timeout Sets the timeout of a get- or put operation in seconds. Defaults to 0 which means no timeout. This setting sets the time the get- or put-operation waits even if there is no activity generated by the underlying operation.
wait Sets the wait time on a get- or put operation in seconds. Defaults to 0 which means to wait forever. This settings sets the time the get- or put operation waits on a completetion of the underlying operation independant upon if it generates activity or not. It is only recommended to use this option if you have special needs for the operation to end if not completed within a certain time. To end operation if nothing happens, please use the timeout setting instead.
The methods returns an instantiated class.
Probe remote location of its capabiliites.
Input parameteres: none. All is taken from already setup paramteres.
Returns 1 if successful in probing. 0 if it failed. Please check the error()-method for more information upon failure.
If successful the probe()-method will update the parameter set of the class based upon what it learned.
We recommend all users call this method prior to other methods.
Sets or gets the name of the Store.
If not input is set, it returns the name of the store. If one add a SCALAR input, that input is set as the name of the store.
Returns 1 when setting the store name or it returns the name the store already has.
Opens a store to execute operations on it.
Needed input depends on the Store-class in question and this method is to be overridden by the inheriting class.
Please call the paramsRequired()-method to find which parameters are needed for a specific Store-class.
The method checks that all required parameters are present and that they meet their regex pattern as defined in the Parameter-class (see Parameter-class for more information).
When all criterias have been met for the required parameters, it will attempt to set the value passed on to open as parameters into the Store-class in question, both required and non-required. It will also check the regex-compliance of non-required parameters. In addition, if the parameter in question has been tagget as a sandbox-parameter, this method will strip away all "..", all slashes and all relevant shell metacharacters to force the parameter to relate only to a filename. The filename will be prepended with the location path of the sandbox-folder from the sandbox-option to the class. This makes it possible to refer to local files, while retaining security for files outside of the sandboxed folder.
The method returns 1 upon successful opening of the store or 0 upon failure. Please check the error()-method upon failure.
Defines a store's get-, put-, del- and extra-parameters that are needed for the store-class in question. The extra-parameters are not part of any command that is run and their defined order does not matter. They are only used to modify the behaviour of the get- og put-parameters. But, all parameters names between the get-,put-, del-command and the extra-parameters are unique. That means that no parameters should be used in two places (get-,put-,del- and/or extra) without having the same meaning.
It is important that all non-private parameters (see the flags in the Parameter-class) are properly escaped or checked for security issues. If a parameter is marked as "private" the Store-class will use quotemeta to escape meta characters. This will in most cases suffice, but there are some command line utilities that are not always happy to get parameters quotes (such as remote- and local paramters). Examples of utilities that are not always happy in some scenarios are sftp and ftp. It is generally considered safe to not quote the local parameter since that in the AURORA-system is supplied by the system and not by the user. However, all non-private parameters should be escaped to be sure there are no issues if possible.
Only non-private parameters can be overwritten by supplying parameters to the open()-method. Others cannot be overwritten by the user of Store-classes.
This method is to be overridden by the inheriting class and are not to be called by user himself.
Several parameters are reserved and has special meaning to the Store-class(es):
local Defines the local data location (for get- and put-operations).
remote Defines the remote data location (for get- and put-operations).
privatekey Defines the privatekey to be used for classes that uses key-authentication (authmode must be set to AUTH_KEY).
privatekeyfile Defines the location of the privatekey file and is to be used for classes that uses key-authentication (authmode must be set to AUTH_KEYFILE).
pw Defines the password to be used for classes that uses password-authentication (authmode must be set to AUTH_PW).
pwfile Defines the location and name of the password file to be used for classes that uses password-authentication (authmode must be set to AUTH_PWFILE).
Returns 1 if successful, 0 if not.
Creates the put-, get- and del-instances of the StoreProcess-class or subclasses. Placeholder to be overridden by inheriting class.
Returns 1 if successful, 0 if not.
This method is not to be called directly by user.
Closes the store and cleans up.
This method closes the store and sets the result of isOpen()-method to 0. It cleans up whatever needs to be cleaned up. This method is to be overridden by the inheriting class if one needs to do some kind of cleanup.
The method is to be called by the user of the class when he is finished using the instance in question to ensure cleanup of temporary resources. It should even be called in event of a failure to perform methods when the user is fininshed with the instance.
Returns 1 upon success, 0 upon failure. Pleace check the error()-method for more information upon failure.
Get or set a GET parameter value.
Input is the name of the parameters to get. Additionally one can add a value as the next parameter if one wishes to set the value of the parameter name given as the first parameter.
Returns the value of the parameter upon get, or upon a set returns the result of setting that parameter on the Parameter-class (see Parameter-class for more information).
Get or set a PUT parameter value.
Input is the name of the parameters to get. Additionally one can add a value as the next parameter if one wishes to set the value of the parameter name given as the first parameter.
Returns the value of the parameter upon get, or upon a set returns the result of setting that parameter on the Parameter-class (see Parameter-class for more information).
Get or set a DEL parameter value.
Input is the name of the parameters to get. Additionally one can add a value as the next parameter if one wishes to set the value of the parameter name given as the first parameter.
Returns the value of the parameter upon get, or upon a set returns the result of setting that parameter on the Parameter-class (see Parameter-class for more information).
Sets a given parameter in all of the GET-, PUT- and DEL-parameters of the Store-class.
Input is the name and the value of the parameter to set.
Returns 1 upon successful set in one or more of the GET-, PUT- or DEL-parameters or 0 upon failure.
Returns the required parameters of the Store-class.
This method get the required parameters of GET-, PUT- and DEL-methods and the extra-parameters of the Store-class.
No input is accepted and it returns a HASH upon success or 0 upon failure. Check the error()-method for more information upon failure.
The return HASH of the required parameters has the following format:
(
paramA => { name => SCALAR (this is a repeat of paramA),
value => SCALAR,
check => REGEX
required => BOOLEAN,
escape => BOOLEAN,
}
paramB => { name => SCALAR (this is a repeat of paramB),
etc...
)For more information on the data returned in the HASH see the Parameter-class documentation.
Starts the method of putting data into store.
The store must first have been opened for this to start successfully. The store cannot already be in a get-, put- or del-mode.
No input is accepted.
It returns 1 upon successful start of the put-operation, 0 upon failure. Please check the error()-method for more information upon failure.
Starts the method of getting data from the store.
The store must first have been opened for this to start successfully. The store cannot already be in a get-, put- or del-mode.
It returns 1 upon successful start of the get-operation, 0 upon failure. Please check the error()-method for more information upon failure.
Starts the method of deleting data from the remote store.
This method is meant to be overridden by inheriting class.
The store must first have been opened for this to start successfully. The store cannot already be in a get-, put- or del-mode.
It returns 1 upon successful start of the del-operation, 0 upon failure. Please check the error()-method for more information upon failure.
It only deletes remotely.
Get the size of the data in the remote location.
This method is to be overridden by the inheriting class.
This method is not to be called by user, but by internal methods.
Returns the size in bytes or undef upon error. Please check the error()-method for more information in the event of an error.
Get the size of the data in the local location.
This method is always calculating on a local filesystem. It is therefore written a general localSize-method that uses the rsync utility. However, this method can be overridden by the inheriting class.
It is recommended that the localSize()-method be careful with how to calculate its own local size, since the local location might have data from several locations/stores. This is not required, however, and the default localSize()-method does not implement this.
Returns the local size in bytes or undef upon error. Please check the error()-method for more information in the event of an error.
Lists the folder/store contents on the remote end.
Input is the path to list. This is optional and default to a blank string.
The function is not recursive and only lists the Store-structure in the path specified. This method is to be overridden by the inheriting class.
Returns a HASH-reference to a structure which must be of the following format upon success:
(
TYPEa => { NAMEa => { NAME => SCALAR, # name of item
TYPE => SCALAR, # either F (File) or D (Folder)
SIZE => SCALAR, # in Bytes
DATETIME => SCALAR, # in unixtime
},
NAMEb => { NAME => SCALAR,
TYPE => SCALAR,
SIZE => SCALAR,
DATETIME => SCALAR,
},
},
TYPEb => { etc...
},
)Type in the initial key also means either F (file), D (folder) or L (link/symbolic). Some information is also repeated in the item itself for ease of use.
Upon failure returns undef. Please check the error()-method for more information upon an error.
Lists the folder/store contents locally (local parameter).
Input is the path to list. The local parameter is added to this path-string. This is optional and default to a blank string.
The function is not recursive and only lists the Store-structure in the path specified. This method is already implemented here ofr local filesystems stores and does not need to be overridden.
Returns a HASH-reference to a structure which must be of the following format upon success:
(
TYPEa => { NAMEa => { NAME => SCALAR, # name of item
TYPE => SCALAR, # either F (File) or D (Folder)
SIZE => SCALAR, # in Bytes
DATETIME => SCALAR, # in unixtime
},
NAMEb => { NAME => SCALAR,
TYPE => SCALAR,
SIZE => SCALAR,
DATETIME => SCALAR,
},
},
TYPEb => { etc...
},
)Type in the initial key also means either F (file), D (folder) or L (link/symbolic). Some information is also repeated in the item itself for ease of use.
Upon failure returns undef. Please check the error()-method for more information upon an error.
Verifies the operation on the store.
This method is to be overridden by the inheriting class, but it needs to verify that the put- or get- operation has been successfully performed and that the data is intact. If not verify method can be implemented it is to return 1 (success).
Returns 1 upon success, 0 upon failure. Check the error()-method for more information upon failure.
Gets the log of either the get-, put- or del-operation depending upon the mode of the Store.
It returns the log-instance of a SmartLog-class that belongs to either the get-, put- or del-StoreProcess operation.
Return the parameter group instance of the get-command for the store.
Returns a reference to a Parameter-class og subclass instance.
Return the parameter group instance of the put-command for the store.
Returns a reference to a Parameter-class og subclass instance.
Return the parameter group instance of the delete-command for the store.
Returns a reference to a Parameter-class og subclass instance.
Returns the exitcode of either the GET-, PUT- or DEL-StoreProcess instance (depends on store-mode). The exitcode is harvested from the executed command in the StoreProcess-class when it returns. See the StoreProcess-class for more information.
Returns the exitcode upon success, undef upon failure. Please check the error()-method for more information upon failure.
Gets or sets the timeout of the GET-, PUT- or DEL-StoreProcess instances (it is the same for all in every case).
Input is the timeout value upon a set. No input accepted upon get.
It returns 1 upon successful set, the timeout value upon successful get.
See the StoreProcess-class for more information on the timeout-option.
Set or get the Store's metadata.
One input accepted: metadata. HASH-reference. Optional (get) and Required (set). If set to undef or not a HASH-reference when set, it will fail with undef in return.
Will return the metadata if no input is specified, it will set the metadata if the metadata-parameter is set and it is a HASH-reference (and then return the set HASH-reference). It will return undef upon failure. Check the error()-method for more information upon a failure.
Gets or sets the wait time of the GET-, PUT- or DEL-StoreProcess instances (it is the same for all in every case).
Input is the wait time value upon a set. No input accepted on get.
It returns 1 upon successful set, the wait value upon successful get.
See the StoreProcess-class for more information on the wait-option.
Returns if the Store-instance is running either a GET-, PUT- or DEL-operation (depends on the mode of the store).
No input accepted.
Returns 1 if a store operation is running, 0 if no operation running or that no operation has started yet.
Returns if the store has been opened or not?
Returns 1 if it has been opened, 0 if not.
Returns the current mode that the store is in.
The possible modes are:
NOP = 0
GET = 1
PUT = 2
DEL = 3These modes can also be referenced by the variables STORE_MODE_NOP, STORE_MODE_GET, STORE_MODE_PUT and STORE_MODE_DEL.
Returns the latest timestamp from any activity of the store-process (either get-, put- or del dependent upon mode).
No input is accepted.
Upon success returns the time in unixtime without locale. Upon failure returns undef. Please check the error()-method for more information upon failure. Typical reason for failure is that neither get-, put- or del- process has been started or are running anymore.
Returns the success of the last GET-, PUT- or DEL-operation.
No input is accepted.
Returns 1 if the last operation was successful, 0 if not or if the store was not opened or no operation run yet.
Upon a 0, one can check the error()-method for more information (if any).
Stops either a GET-, PUT- or DEL-operation on a store (depends on mode).
No input is accepted.
Returns 1 upon successfully ceasing operation of a GET-, PUT- or DEL-operation, 0 if not.
It will also return 0 if store is not opened yet or no GET-, PUT- or DEL-operation is running. In such cases one can check the error()-method for more information.
Returns the last error of the store (if any).
No input is accepted.
Return value is a SCALAR.