pyfora.aws¶

class pyfora.aws.Cluster(name, region)¶

A cluster of pyfora instances in Amazon Web Services EC2.

A Cluster object can be used to launch clusters, list instances in a cluster, add instances to an existing cluster, or shut down running clusters.

Parameters:	name (str) – A name to identify the cluster region (str) – An AWS region for the cluster (e.g. us-west-2, ap-southeast-1). Defaults to ‘us-east-1’.

add_workers(manager, num_workers, spot_price=None, callback=None)¶

Adds workers to an existing cluster.

The EC2 instance type used launch the added workers is the same as that of the already launched instances.

Parameters:

manager (boto.ec2.instance) – The cluster’s manager. It can be retrieved by calling Cluster.list_instances().
num_workers (int) – The number of workers to add.
spot_price (float, optional) – If specified, launch the cluster using EC2 spot instances with the specified max bid price.
callback (callable, optional) – An optional callback that receives progress notifications during the launch process. The callable should accept a single argument of type ClusterEvent.

Returns (list):: A list of boto.ec2.instance objects representing the launched workers.

launch(instance_type, ssh_keyname, num_instances=1, open_public_port=False, vpc_id=None, subnet_id=None, security_group_id=None, spot_price=None, callback=None)¶

Launches a new cluster in EC2.

Instances are launched from a vanilla Ubuntu image, docker and other dependencies are installed, and the ufora/service image is pulled and started.

If launching a single instance, it is configured to run both the pyfora mangaer and worker. Additional instances only run the worker and are configured to connect to the cluster’s manager.

Parameters:	instance_type (`str`) – The EC2 instance type to use (e.g. c3.4xlarge, m4.large, etc.) ssh_keyname (`str`) – The name of an SSH key-pair in EC2. Instances are launched with that key and you MUST have its private key in order to SSH into them. num_instances (`int`, optional) – The number of instances to launch. Defaults to 1. open_public_port (`bool`, optional) – Whether the pyfora HTTP port should be open for access over the internet. Defaults to False. If False, you can only connect to the cluster from with EC2 or by tunnelling HTTP over SSH. vpc_id (`str`, optional) – The id of an EC2 Virtual Private Cloud (VPC) into which the instances are launched. Attempt to launch to EC2 Classic if omitted. subnet_id (`str`, optional) – If using vpc_id, this is the ID of the VPC subnet to launch the instances into. security_group_id (`str`, optional) – The ID of an EC2 Security Group to launch the instances into. If omitted, a new security group called “pyfora” is created. spot_price (`float`, optional) – If specified, launch the cluster using EC2 spot instances with the specified max bid price. callback (`callable`, optional) – An optional callback that receives progress notifications during the launch process. The callable should accept a single argument of type `ClusterEvent`.
Returns:	The collection of instances in the newly created cluster.
Return type:	`Instances`

list_instances()¶

Returns the current instances in the cluster.

Returns:	The collection of instances in the cluster.
Return type:	`Instances`

stop(instances=None, terminate=False)¶

Stops or terminates instances.

Parameters:	instances (`list`, optional) – a list of `boto.ec2.instance` objects representing the instances to be stopped. If omitted, all instances in the cluster are stopped. terminate (`bool`, optional) – If True, instances are terminated. Otherwise they are stopped. On clusters of spot instances, this argument must be set to True because spot instances cannot be stopped, only terminated.

class pyfora.aws.Cluster.Instances¶

Collection of instances in a cluster.

manager¶: list – a list (normally of length 1) of boto.ec2.instance object(s) representing the EC2 instance running the cluster’s manager.

workers¶: list – a list (possibly empty) of boto.ec2.instance objects representing the EC2 instances running the cluster’s workers. Note: The worker running on the manager instances is not included in this list.

unfulfilled¶: list – a list of zero or more boto.ec2.SpotInstanceRequest objects representing spot instance requests that have not been fulfilled due to price or availability.

class pyfora.aws.Cluster.ClusterEvent¶

An object representing an event emitted during calls to Cluster methods.

event_type¶: EventTypes – The type of event.

body¶

The content of the event, which varies by event type:

EventTypes.Launching, EventTypes.LaunchFailed:

Either “manager” or “worker”

EventTypes.WaitingForServices, EventTypes.Done, EventTypes.Failed:

None

EventTypes.Launched:: A tuple with two elements. The first is a string whose value is either “manager” or “worker”, and the second is a list of boto.ec2.instance objects.
EventTypes.InstanceStatus:: a dict whose keys are strings that identify various statuses an instances can be in (e.g. ‘pending’, ‘ready’, ‘installing dependencies’, etc.), and the values are lists of boto.ec2.instance objects representing all the instances that are in that state.

class pyfora.aws.Cluster.EventTypes¶

An enumeration of the types of events that can be emitted during calls to Cluster methods.

Launching¶: One or more instances are about to be launched.

Launched¶: One or more instances have been launched.

LaunchFailed¶: One or more instances failed to launch.

InstanceStatus¶: An event with detailed progress information.

WaitingForServices¶: Starting to wait for all post-launch steps to complete and for the pyfora service to start.

Done¶: Operation completed successfully.

Failed¶: Operation failed.