SyncPlanet API

The synchronization « tracker » consists in a database to store the files' states along with an HTTP web service answering to a specific set of operations. You can either access:

• your dedicated tracker, usually accessible behind a domain like https://<trackername>.syncplanet.io,
• the main server at https://syncplanet.io,
• a demo server explaining basics at https://demo.damas.io

The tracker is based on the free and open source project damas-core (data management system core) to store the objects needed by the synchronization system and to provide basic CRUD operations to interact with them.

You can visit the Environment Setup Guide to setup a Python, Javascript or Shell scripting environment.

Objects Types

The database stores objects using the JSON format and are indexed using the _id key being unique identifiers Strings.

Files

The files are indexed using their path, as a unique identifier, under the _id key. A few other keys are used to control the synchronization system (see the list below). Any number of custom keys can be added without limitation, this might be useful for an integration in a custom asset management system.

File Publish

When some files are created and need to be propagated on several sites, at publish time, a JSON object is provided, so the synchronization service can process the file specified by the _id key:

{
  "_id": "/file/path",         /* UNIX path format (String) */
  "comment": "<text>",         /* Text comment of user when published (String) */
  "origin": "<sitename>",      /* Volume name where the file is coming from  (String)*/
  "sync_priority": 7,          /* Positive or negative, 0 is default normal priority (Number) */ 
  "syncEverywhere": true       /* overide default behavior in globalParameters (Boolean) */
}

_id (String)

File's unique identifier, using UNIX path format.

comment (String)

User's text comment, specified at publish.

origin (String)

Site's name, on which the file was created, and has to be retrieved from.

sync_priority (Number)

Order preference to process this file. The files with a higher sync_priority number are processed first, default is 0 (normal priority) and it can be negative.

syncEverywhere (Boolean)

Does this file has to be propagated everywhere? (on every other site that is was originated from?)

{
    "syncEverywhere": False,
    "sync_studio1": True,
    "sync_studio2": False,
    "sync_studio3": True
}

your_key (String, Number, Array, Boolean, Object)

New Keys at Creation

The author and time keys are automatically inserted by the tracker when a new file is created:

{
  "_id": "/file/path",
  [...]
  "author": "<@username>",  /* String, authenticated user who did the publish */
  "time": 1567529676000     /* Number, time of the element at creation, milliseconds EPOCH */
}

author (String)

The authenticated user name who did the publish.

time (Number)

The time at publish. It is used to display and sort the elements in the interfaces. Epoch milliseconds: Number of milliseconds since January 1st 1970.

New Keys at Process Time

Some keys are created and updated as the synchronization process runs. These keys are useful for the monitoring interfaces, to get the file's history log, or to debug the file's synchronization:

{
  "_id": "/file/path",
  [...]
  "file_mtime": 1567529676000,      /* Number,  modification time on the origin file system (epoch time in milliseconds) */
  "file_size": 1286230019,          /* Number,  number of bytes on the origin file system */
  "modified_site5": 15676534821582, /* Number,  a modification was scanned on site5 */
  "syncError_site2": 23,            /* Number,  error encountered on site2 (rsync exit status code) */
  "syncLastAttempt_site2": 1567516933000, /* Number, time of last failed attempt on site2 */
  "synced_site1": 15675564912347,   /* Number,  successful delivery time on site1 */
  "synced_site3": 15675564591231,   /* Number,  successful delivery time on site3 */
  "synced_site4": 15675565175645,   /* Number,  successful delivery time on site4 */
  "syncing_site6": 1754897397000    /* the service is propagating this file on site6 since epoch (Number) */
}

• file_mtime the value of the mtime attribute of the file in the file system (unix timestamp - epoch - in milliseconds)
• file_size the value of the size attribute of the file in the file system (size in bytes)
• modified_sitename unix timestamp in milliseconds
• syncError_sitename exit status code if an error occured on that site. 23 is: file not found
• syncLastAttempt_sitename: time of the last unsuccessful sync attempt for sitename. unix timestamp in milliseconds
• synced_<sitename>: time of the successful sync for sitename. unix timestamp in milliseconds
• syncing_<sitename>: the service is processing this file for this site since specified epoch

Optional Keys

If you need a multi-site locking system please contat us.

{
  "lock": "<@username>"            /* String,  username who is editing the file */
}

Users

The users are described as elements wearing some reserved keys: username, password, class plus optional keys.

{
"_id": "@username",               /* String, unique user name identifying the user in the database */
"class": "editor",                /* String, a user class for permissions. Could be guest, user, editor, admin  */  
"email": "usermail@address.com",  /* String */
"fullname": "Firstname Lastname", /* String */
"password": "13d3a2a16c0cd2f7bf115d471999377e" /* String, hashed password */
}

The Authentication page gives more details about the authentication mechanism.

Generic function lists and objects

CRUD Function List

A set of generic CRUD functions is provided to create, read, modify and delete elements:

• create - insert element(s) in the database
• read - retrieve element(s) using identifier(s)
• update - modify keys on existing element(s)
• upsert - create or modify element(s)
• delete - delete element(s)

User Management

• signIn- request an authentication token from the server
• signOut - revoke a token
• verify - ask the server for the authentication status and user

Search queries

• graph - retrieve connected edges and nodes (recursive)
• search - find elements matching a query string
• search_one - find first element matching a query string
• search_mongo - find elements using a MongoDB query object (if MongoDB is the back-end)

Function Details

lock

Lock file(s) for edition. Sets a lock key on elements with current authenticated username as value)

lock( identifiers [, callback] )

parameters

• identifiers {string | array} : identifier(s) string to lock
• callback {function} (js only, optional) : if specified, the request is asynchronous

return values

• returns true on success, false otherwise
• returns false if the element is already locked

# Python
# lock one asset
project.lock('/project/path/to/file')
# True

# lock multiple assets in 1 request
project.lock(['/project/path/to/file1', '/project/another_file_path'])
# True

publish

Add files to the index

publish( elements [, callback] )

Many options are provided at publish time, allowing Folder Packages and Destination Filtering.

parameters

• elements {object | array} : object(s) to insert in the database
• callback {functon} (js only, optional) : if specified, the request is asynchronous

return values

• returns an array of nodes (containing parent nodes and child nodes) on success
• returns null (Javascript) or None (Python) on failure

Same specifications as /api/create, except that it is expecting specific keys and a child element is created to keep track of the states upon publish

When we publish a file that is already tracked, the function returns an error

examples

{
  "_id": "/project/path/new_file",
  "comment": "text",
  "origin": "sitename"
}

Child node :

{
  "_id": "55ae0b1ed81e88357d77d0e9",
  "#parent" : "/project/path/new_file",
  "comment": "text",
  "origin": "sitename"
}

• _id path or an array of paths
• key origin alphanumerical name without space, for ease of use. For multi-site.

optional keys (these keys are not mandatory but could ease multi sites configurations and version control):

• file_mtime Number (milliseconds since 1 Jan 1970 00:00)
• file_size Number (number of bytes)
• version Number

In a multi-site environment, the origin and _id path are used to retrieve the file from the source server.

unlock

Unlock a locked asset

unlock ( identifiers [, callback] )

parameters

• identifiers {string | array} : identifier(s) string(s) to lock
• callback {function} (js only, optional) : if specified, the request is asynchronous

return values

• returns a boolean or an array of booleans (depending on the input)

If the asset is not locked or locked for someone else (lock key value != authenticated user name) it returns false. If it was successfully unlocked, returns true.

comment

Add a comment to one or several element(s)

comment ( elements [, callback] )

parameters

• elements {object | array} : object(s) specifying the elements identifiers and comment string
• callback {function} (js only, optional) : if specified, the request is asynchronous

return values

• returns an element or an array of elements on success
• null (Javascript) or None (Python) on failure

Sets author key on elements with the authenticated username as value, as well as time key with the current time at creation.

# Python
# single parent id
>>> project.comment({"#parent" : "asset_id", "comment" : "text"})
{u'author' : u'username', u'time' : 1480588505449, u'#parent' : u'asset_id', u'comment' : u'text'}

# multiple parent ids
>>> project.comment({"#parent" : ["asset_id1", "asset_id2"], "comment" : "text"})
[{u'author' : u'username', u'time' : 1480588505449, u'#parent' : u'asset_id1', u'comment' : u'text'}, {u'author' : u'username', u'time' : 1480588505449, u'#parent' : u'asset_id2', u'comment' : u'text'}]

// Javascript
damas.comment({'#parent' : "asset_id", comment : "text"});
>> Object { author: "damas", time: 1480588505449, '#parent': "asset_id", comment: "text" }

// comment using an asynchronous call
damas.comment({'#parent' : "asset_id", comment : "text"}, function (node) {
    // asynchronous mode
    console.log(node.time);
});

SyncPlanet-specific older documentation

Global Parameters

The global configuration parameters are accessible through a JSON element that is used by the synchronization agents.

{
"_id": "rsyncGlobalParameters", /* generic parameter */
"enableScans": true,            /* parameter for scans */
"excludePatternRules": "",      /* parameter for scans */
"excludedFiles":       "",      /* parameter for scans */
"ignoreExistingFiles": false,   /* parameter for scans */
"includePatternRules": "",      /* parameter for scans */
"rsaPublicKey":        "",      /* key to install under volumes' .ssh/authorized_keys */
"sleepBetweenScans": 60,        /* parameter for scans */
"syncEverywhere": true,         /* [Destination Filtering](Destination-Filtering) */
"time": 1555711910004           /* generic parameter */
}

Volumes

Each volume is described as an element which holds its configuration and options to control its behavior.

{
"_id": "sit/inthebox/mice/amopix",  /* the volume identifier */
"agentStartTime": 1599731439000,    /* agent time when launched */
"agentStatus": "on",                /* status of the agent, on / off / undefined */
"agentVersion": "1.0.5",            /* version of the running agent */
"author": "support",                /* generic parameter */
"enableDiscovery": true,             /* Boolean, filesystem scan switch */
"enableDiscoveryPackages": true,     /* Boolean, filesystem scan switch */
"enableDiscoveryUpdatedFiles": true, /* Boolean, filesystem scan switch */
"enableDiscoveryDeletedFiles": true, /* Boolean, filesystem scan switch */
"enableEmission": true,              /* Boolean, file emission switch */
"enableReception": true,             /* Boolean, file reception switch */
"error": 0,                    /* Number, error encountered */
"exit_emission": 0,            /* Number, error encountered */ 
"exit_reception": 0,           /* Number, error encountered */ 
"exit_scan": 0,                /* Number, error encountered */ 
"last_emission": 1571918493000,   /* Number, last emission */
"last_reception": 1571910212000,  /* Number, last reception */
"last_scan": 1571918421000,       /* Number, last filesystem scan */
"lastConnectionAttempt": 1600942773000,      /* Number, time of last connection attempt */
"lastConnectionStatusChange": 1600512139000, /* Number, time of previous status */
"lastConnectionSuccess": 1600942773000,      /* Number, last successful connection to the volume */
"last_emission": 1600942573000,   /* Number, last file emission time */
"last_reception": 1598770303000,  /* Number, last file reception time */
"last_scan": 1600942965000,       /* Number, last volume scan time */
"name": "amopix",             /* String, volume's name */
"notifiedEmails": ""          /* String, comma separated email addresses */
"rsync_args": "",             /* String, arguments for rsync */
"rsync_env": "",              /* String, environment for rsync */
"scan_duration": 14,          /* Number, last filesystem scan duration */
"stderr_emission": "",        /* String, stderr if emission error */
"stderr_scan": "",            /* String, stderr if scan error */
"syncAmountEmission":  10,    /* Number, Number of elements uploaded each time */
"syncAmountReception": 10,    /* Number, Number of elements downloaded each time */
"time": 1563285980221,        /* generic parameter */
"url": "ssh://user@adress:port/project/" /* String, URL of the volume */
}