Quick Start¶
We like to test Xenon against a Docker image: nlesc/xenon-slurm. If you have docker all setup, you can run this image as follows:
$ docker pull nlesc/xenon-slurm
...
$ docker run --detach --publish 10022:22 nlesc/xenon-slurm
Try logging onto this image by ssh, to make sure everything works. The username is xenon, the password javagat:
$ ssh localhost -p 10022 -l xenon
xenon@localhost's password: <javagat>
$ exit
Connection to localhost closed.
Starting the server¶
To get anything done in PyXenon, we need to start the GRPC server:
import xenon
xenon.init()
Writing to a remote filesystem¶
Next, let’s try to copy a file to the container. We need credentials to access anything on the remote side.
from xenon import PasswordCredential, FileSystem
credential = PasswordCredential(
username='xenon',
password='javagat')
remotefs = FileSystem.create(
'sftp', location='localhost:10022',
password_credential=credential)
We can write to a file by streaming. The second argument to
write_to_file()
should be an iterable. It will be read in a separate
thread, so it is allowed to be blocking. Here we’ll do nothing so fancy:
from xenon import Path
target = Path('hello.sh')
if remotefs.exists(target):
remotefs.delete(target)
remotefs.write_to_file(
target,
[b'#!/bin/sh\n',
b'echo "Hello, World!"\n'])
Running a script¶
The remote machine runs a SLURM job scheduler. We describe a job in a
JobDescription
object. This seems a bit long-winded, but in
practice you’ll be reusing the descriptions a lot.
from xenon import Scheduler, JobDescription
scheduler = Scheduler.create(
adaptor='slurm',
location='ssh://localhost:10022',
password_credential=credential)
job_description = JobDescription(
executable='/bin/sh',
arguments=['hello.sh'],
stdout='result.txt')
job = scheduler.submit_batch_job(job_description)
state = scheduler.wait_until_done(job)
print(state)
Retrieving the result¶
Just as we can write data by sending an iterable, we can read data from a file and recieve a generator yielding bytes objects. Here we realize the transfer by joining the data chunks into a string:
text = ''.join(chunk.decode() for chunk in
remotefs.read_from_file(Path('result.txt')))
print(text)