Choosing to put your data products in a long-term repository can:
SESYNC provides access to a remote RStudio session, via a web browser, in order to work in R while directly connected to other SESYNC resources (file storage, databases, the cluster, etc).
When you submit your travel planner to our travel office, please make sure to include a list of all remote participants that will be joining your meeting. You may add participants after the 8 week deadline, but we require that you notify us of all participants at least one week before the start of your meeting.
The submission script lists the shell commands to be run from the cluster. To run Python code, the submission script can be as simple as:
You can use SESYNC’s RStudio Server to submit jobs to our cluster and view the resulting output. In the following example, you will learn how to:
To publish a R shiny application on the SESYNC server, your files will need to be copied from your working directory to the shiny-apps-data shared folder (/nfs/shiny-apps-data on RStudio Server). Please contact SESYNC IT staff if you would like to host an app on SESYNC’s Shiny Server.
SESYNC offers private git hosting through our GitLab server at . When you connect to gitlab using your sesync name and password, you’ll see a dashboard of recent activity on projects that you are part of. If this is your first time connecting, it may be a little quiet.
You can connect to the cluster through our ssh gateway service running at ssh.sesync.org or by submitting jobs through rstudio. The workflow for using a cluster is a little bit different from a typical run in R or python. In addition to your processing code, you must give the cluster a list of execution instructions and a description of the resources your analysis will require. Any output from your script will be written out to a file called slurm-[jobID].out and errors go to slurm-[jobID].err.
SESYNC researchers and staff can provide a link for external collaborators to upload and/or download data from your research project directory
You can upload and download data from your research data directory using an SFTP client. We recommend FileZilla
The data storage available at SESYNC has multiple points of access, suitable for different methods of transferring and accessing project data.
Download a high-level overview for general reference, or see the FAQ.
SESYNC has installed the eBeam whitebaord capture software on all of our conference room PC’s and laptops.
SESYNC Windows client virtual machines are setup to use dynamic memory. What this means is that your virtual machine will show a different amount of memory available based on its current usage. You still have access to the full amount of memory allocated if needed. The virtual machine will grab more memory from the hypervisor when needed automatically.
Due to some quirks on our storage system your git repo may show that all of your files have modifications. If you perform a ‘git diff’ you will see a list that looks like:
A virtual machine is a Windows or Linux machine that runs on and shares computing resources with a physical machine known as a hypervisor. Virtual machines allow the deployment of multiple machines or services on one or several hypervisors to better utilize computing resources (CPU cores, memory, etc…)
SESYNC has the ability to deploy custom Windows and Linux Virtual Machines for use by groups. If there is a software or service needed that is not provided by our shared infrastructure, we can deploy a virtual machine to meet your needs.
SESYNC can work with your group to obtain software that would be beneficial to your research while at SESYNC. All software purchased by SESYNC must be installed on SESYNC-owned equipment. We can provide virtual machines to use by your group members to access your purchased software for use.
When are the server maintenance windows?
We’re sensitive to the fact that your jobs may need to run over our maintenance window and will take a reasonable effort to ensure they aren’t disrupted. In order to ensure as minimal disruption as possible, these are the steps that we take:
SESYNC uses the sympa mailing list server to manage mailing lists. You can manage your list by going to lists.sesync.org and entering your e-mail address and SESYNC password in the upper right.
Shiny applications hosted by SESYNC are publically available at the URL http://shiny.sesync.org/apps/<APPNAME>, where <APPNAME> is unique for each app. To get started on publishing an app, read the quick start guide.
SESYNC linux resources are deployed on a private network at SESYNC and are accessed via our ssh gateway at ssh.sesync.org.
SESYNC offers the ability for your group to schedule conference calls outside of your onsite meetings. These calls can either be dial-in only, or audio-video. To schedule a call, please contact SESYNC it staff firstname.lastname@example.org at least 7 days in advance of your call and we will configure a one time or standing meeting for you. Please let your participants know to keep a lookout for a meeting invitation from SESYNC’s IT staff. After you schedule your call, the following will happen:
SESYNC provides a large, shared file storage for hosting data for all of our projects. Once our IT staff creates a data storage location, its available to access on any Linux or Windows your group has access to. For SESYNC employees and postdocs you can directly access this storage from your SESYNC desktop or laptop. For all others, we provide a web gateway to your groups’ data storage.
For new groups, we generally follow the timeline below:
If you already have projects on GitHub that you are working on, we prefer that you continue to use GitHub due to its open nature. We’ll gladly push and pull code from your public repository. We provide GitLab locally for projects that are just starting up, have sensitive data, or are not quite mature enough to be pushed out into the world.
The three are often a source of confusion.
Yes! You can push a local git repository to any new remote resource. Please note that only your source code will move. However, the additional features you use (e.g. wiki, issues, etc.) will need to be manually copied.
SESYNC’s computational cluster enables users to run medium-to-large scale analyses by distributing multiple, independent tasks across many computers. This setup is ideal for tasks that require applying the same algorithm or a parameter set over independent units in a large data set.
SESYNC provides remote access to all desktop resources through a browser based RDP gateway. Please point your web browser to https://desktop.sesync.org to access your virtual desktop
We highly recommend using the scheduled cluster for running all of your CPU-intensive or long running programs. Below is SESYNC policy for long running processes on our different types of resources:
Access to a MySQL or PostgreSQL database from within R requires a database client and connection parameters (including a password) for the particular database you wish to access. This FAQ explains a recommended configuration that allows R to access a database on the SESYNC infrastructure. The instructions below are specifically for a PostgreSQL database, but analagous instructions apply to MySQL.
When you submit your travel planner to our travel office, please make sure to include a list of all remote participants that will be joining your meeting. You may add participants after the 8 week deadline but we require that you notify us of all participants at least one week befor ethe start of the meeting.
SESYNC has five conference rooms and a large breakout space equipped with HD screens, projectors, whiteboards and an array of collaborative tools. Please note, depending on the size and requests of your group, your assigned meeting room may have a different table arrangement that waht is show in the pictures below.
Users can sync their project data to your local machine using the PydioSync client.
Anyone who is a PI on a science team, or is part of a long running pursuit, will receive an email with instructions on completing account setup prior to their first meeting.
RStudio projects are folders that contain project files and a special .Rproj file. To link an RStudio project with a git repository, follow these steps:
Point your web browser to https://pwm.sesync.org.
A SESYNC username is usually your first initial followed by your last name, (i.e. “John Smith” is jsmith). Common or very long names may not follow this pattern.
SESYNC has an extensive set of computing resources and expertiese available for researchers. Download a high-level overview of all services and support SESYNC offers for general information, or scan the tables below for a quick reference. Direct all questions to email@example.com.
Email firstname.lastname@example.org with your question or support request.
Partway through her LTER Postdoc at SESYNC, ecologist Meghan Avolio ran into trouble manipulating her data on plant communities with dplyr functions. I had encouraged Meghan to modularize her scripts by writing functions for common steps in her pipeline (such as converting count data into rank-abundance curves). “You’ll love writing functions!” I said wrongly.
Many funding agencies require proposals to include a section addressing plans for data management. This includes how you will handle data as it is being collected during the project, as well as plans for sharing and archiving once the project is complete. Here is a collection of resources we’ve found helpful for writing DMPs:
Photos, as a source of data, or to aid in the interpretation of data, can be a useful addition to RShiny applications. Here are two examples of using photo data: one that dispalys images from URLs, and another that uses species names to find pictures of animals.
Speak to a database in its native language.
Acquire data from websites and APIs.
Write formulas for regression in R and Stan.
Tour R packages that make static and interactive maps.
Craft publication-quality graphics with ggplot2.
Perform version control from the command line.
Use spatial data in NetLogo ABMs.
Manipulate geospatial data with open source tools.
Get your data in shape with dplyr.
Build agent-based models with a simple graphical interface.
Learn to use git with GitHub.
Get interactive with the Shiny R package.
Contain an entire pipeline in one document.
Start learning R in RStudio.
Start learning Python with Numpy, Pandas and Matplotlib.
Carve your texts into structured data.
Control agent-based model simulations from R.
Meet the open source stack underlying geospatial data.
Implement open agent-based models.
Make your data safe, scalable and relational.