Cyberhelp

for Researchers & Teams

Zoom for SESYNC Teams

Depending on your interests and needs, SESYNC can do any or all of the following: a) test out meeting settings and consult ahead of time; b) just schedule a meeting and transfer host role at meeting start; c) act as host or co-host throughout one or more meeting sessions to manage participants, breakout rooms, polls, and/or shared content; d) be “on call” during your meeting time for real-time support over chat, phone, or video.

Run Code in Parallel on a Windows Virtual Machine

If you have a program that runs only on Windows systems, and you would like to run the program in parallel, it is not possible to use SESYNC’s Slurm cluster. However, it is possible to take advantage of SESYNC’s Winanalytics virtual machine, which has multiple cores and much more available RAM than the typical laptop. You will need to write a little bit of code in PowerShell, which is Windows’ shell scripting languge and is fairly similar to Bash scripting. Here is a quick walkthrough of how to do this.

The Compute Cluster

SESYNC provides a high-performance computing cluster for memory-intensive and time-intensive computing tasks. (FAQ: What is the SESYNC cluster?) You can connect to the cluster through our ssh gateway service running at ssh.sesync.org or by submitting jobs through RStudio. The workflow for using a cluster is a little bit different from a typical run in R or python. In addition to your processing code, you must give the cluster a list of execution instructions and a description of the resources your analysis will require. Any output from your script will be written out to a file called slurm-[jobID].out and errors go to slurm-[jobID].err.

Command-line Stata (from RStudio Server)

To support existing data analysis pipelines that use the Stata software, SESYNC has purchased a Stata license and created a dedicated virtual machine for remote use by affiliated researchers. This quick start guide explains the essential steps for evaluating Stata commands over SSH or from SESYNC’s RStudio server.

Creating an Email Group List

Navigate to lists.sesync.org in a Web Browser

Connect to a Database Server

This Quick Start guide will walk you through establishing a connection to a database on SESYNC’s server. Access to your pursuit’s relational database management system (RDBMS) requires communication between a server application (PostgreSQL or MySQL) and a client application (RStudio, Jupyter, psql, etc.).

Jupyter Server

SESYNC provides access to remote JupyterLab sessions via a web browser. The Jupyter Project provides an environment for Python development, and SESYNC’s Jupyter Server adds direct connections to resources like shared file storage, databases, GitLab, and a compute cluster.

Publish Synthesis Data Products

Choosing to publish your data products in a long-term repository can:

Share Files and Folders with Anyone

SESYNC researchers and staff can provide a link for external collaborators to upload (donwload) files to (from) any research data directory they can access. We recommend this mechanism for receiving datasets from external collaborators: create a new folder and turn it into a public “file drop”, as described below.

RStudio Server

SESYNC provides access to a remote RStudio session, via a web browser, in order to work in R while directly connected to other SESYNC resources (file storage, databases, the cluster, etc).

Remote Meeting Participants

When you submit your travel planner to our travel office, please make sure to include a list of all remote participants that will be joining your meeting. You may add participants after the 8 week deadline, but we require that you notify us of all participants at least one week before the start of your meeting.

Publish a Shiny App

To publish a R shiny application on the SESYNC server, your files will need to be copied from your working directory to the shiny-apps-data shared folder (/nfs/shiny-apps-data on RStudio Server). Please contact SESYNC IT staff if you would like to host an app on SESYNC’s Shiny Server.

Create Projects on GitLab

SESYNC offers private git hosting through our GitLab server. When you connect to our GitLab Community Edition (CE) instance using your SESYNC username and password, you’ll see a dashboard of recent activity on projects that you are part of. If this is your first time connecting, it may be a little quiet.

Bulk Uploads and Downloads by SFTP

You can upload and download data from your research data directory using an SFTP client. We recommend Cyberduck or WinSCP

Research Data Directory

SESYNC provides a large, shared file store to host data for all projects. Project participants have access to the research data directory for their project from our compute servers, a web portal, a desktop application for syncing, and SSH.

eBeam Whiteboards

SESYNC has installed the eBeam whitebaord capture software on all of our conference room PC’s and laptops.

How can I secure my virtual meeting?

Use of video conference platforms has exploded now that we are all working from home.

What are common options for Slurm jobs, and how do I set them?

There are a few different ways to run a job on SESYNC’s Slurm compute cluster, but all of them ultimately run a command called sbatch to submit the job to the cluster. The sbatch program is part of the Slurm software package and has a lot of different options. These include a maximum length of time your jobs can run, how much memory you are requesting, whether you want to be notified by email when your job finishes running, etc. It’s possible to run a Slurm job without setting any of the options and going with all defaults, but there are times when you might want to customize the options.

How do I work with a git-versioned project in Jupyter Lab?

There’s no user-friendly way to do this. You have to use git in the command line.

How do I run an interactive job on the cluster?

SESYNC’s Slurm compute cluster allows users to run big memory- and processor-intensive jobs. Many users don’t know that you can access the memory and processing power of the cluster interactively, typing commands directly into the command line or into an R or Python session. This FAQ briefly describes how to start an interactive job on the Slurm cluster.

Where should I store temporary files created by Slurm cluster jobs?

Many jobs on the Slurm compute cluster generate lots of big files that require large amounts of memory to be stored but are only needed temporarily. There are two different ways to easily store large temporary files created by cluster jobs: temporary storage on a specific node (/tmp/) and scratch space accessible from all nodes (/nfs/scratch/).

How much data can I store in my research data directory?

TL;DR: Try to have a general idea of your data storage needs, and discuss it with the data science team if you are concerned, but do not be too worried unless you are going well over 1 terabyte.

What resources exist for collaborative writing?

There are several resources available for collaborative writing depending on which platform you prefer to work. These are the resources SESYNC groups have successfully used in the past.

How to create a symlink to a research directory in Jupyter lab?

To access and see your data directory from the side data console, you need to first set a symlink to your research directory. To do this, open a terminal notebook in Jupyter lab. The command to set the symlink is:

What are SESYNC's cyberinfrastructure policies?

Please see the cyberinfrastructre “Process & Policies” page on SESYNC’s main website.

How does SESYNC wind down computational support?

SESYNC’s data storage and computational resources are available to pursuit participants for approximately one year after the final meeting.

Why isn't my research data directory in '/nfs'?

It is, or at least will be as soon as you need it! Any research data directory you have access to will be mounted to the filesystem at “/nfs” when you access it. If you have not touched any of the files in there for a while, it may have un-mounted and appear to be missing. So if you don’t see your “*-data” folder under “/nfs”, just navigate directly to the folder and it will instantly mount. For example, if your research data directory is “cooltrees-data”, then enter the full path as “/nfs/cooltrees-data” in the file browser or from the command line.

How do I connect to a video conference?

If you are a remote participant, see the below instructions the download the Skype for Business pugin and join your meeting.

Why does does my virtual machine show less memory than I requested?

SESYNC Windows client virtual machines are setup to use dynamic memory. What this means is that your virtual machine will show a different amount of memory available based on its current usage. You still have access to the full amount of memory allocated if needed. The virtual machine will grab more memory from the hypervisor when needed automatically.

Why does git show that all my files changed when I didn't change them?

Due to some quirks on our storage system your git repo may show that all of your files have modifications. If you perform a ‘git diff’ you will see a list that looks like:

What is a virtual machine?

A virtual machine is a Windows or Linux machine that runs on and shares computing resources with a physical machine known as a hypervisor. Virtual machines allow the deployment of multiple machines or services on one or several hypervisors to better utilize computing resources (CPU cores, memory, etc…)

What support does SESYNC provide for custom virtual machines?

SESYNC has the ability to deploy custom Windows and Linux Virtual Machines for use by groups. If there is a software or service needed that is not provided by our shared infrastructure, we can deploy a virtual machine to meet your needs.

Will SESYNC purchase software?

SESYNC can work with your group to obtain software that would be beneficial to your research while at SESYNC. All software purchased by SESYNC must be installed on SESYNC-owned equipment. We can provide virtual machines to use by your group members to access your purchased software for use.

When are the server maintenance windows?

When are the server maintenance windows?

What happens to my jobs during the maintenance window?

We’re sensitive to the fact that your jobs may need to run over our maintenance window and will take a reasonable effort to ensure they aren’t disrupted. In order to ensure as minimal disruption as possible, these are the steps that we take:

How do I manage my mailing list?

SESYNC uses the sympa mailing list server to manage mailing lists. You can manage your list by going to lists.sesync.org and entering your e-mail address and SESYNC password in the upper right.

Where are Shiny apps published?

Shiny applications hosted by SESYNC are publically available at the URL http://shiny.sesync.org/apps/<APPNAME>, where <APPNAME> is unique for each app. To get started on publishing an app, read the quick start guide.

How do I access Linux resources?

SESYNC Linux resources are deployed on a private network at SESYNC and are accessed via our ssh gateway at ssh.sesync.org. These resources include RStudio, Jupyter lab, and our compute cluster. Please DO NOT run your computational processing on the ssh gateway, it has limited memory and processing power. Instead, use the ssh gateway to submit jobs to SESYNC’s compute cluster or to connect to your virtual machine.

How do I schedule a conference call?

SESYNC offers the ability for your group to schedule conference calls outside of your onsite meetings. These calls can either be dial-in only, or audio-video. To schedule a call, please contact SESYNC it staff cyberhelp@sesync.org at least 7 days in advance of your call and we will configure a one time or standing meeting for you. Please let your participants know to keep a lookout for a meeting invitation from SESYNC’s IT staff. After you schedule your call, the following will happen:

How do I access my research data directory?

Navigate to https://files.sesync.org and log in with your SESYNC username and password. The folders listed under “External storages” are each a shared research data directory accessble to participants in the corresponding project.

When does SESYNC setup resources for my group?

For new groups, we generally follow the timeline below:

Should I use GitHub or SESYNC's GitLab?

If you already have projects on GitHub that you are working on, we prefer that you continue to use GitHub due to its open nature. We’ll gladly push and pull code from your public repository. We provide GitLab locally for projects that are just starting up, have sensitive data, or are not quite mature enough to be pushed out into the world.

What's the difference between git, GitHub, and GitLab?

The three are often a source of confusion.

Can code move between GitLab, GitHub, and Bitbucket?

Yes! You can push a local git repository to any new remote resource. Please note that only your source code will move. However, the additional features you use (e.g. wiki, issues, etc.) will need to be manually copied.

What is the compute cluster?

SESYNC’s computational cluster (see quickstart page) enables users to run medium-to-large scale analyses by distributing multiple, independent tasks across many computers. This setup is ideal for tasks that require applying the same algorithm or a parameter set over independent units in a large data set.

How do I access my Windows virtual machine?

SESYNC provides remote access to all desktop resources through a browser based Remote Desktop Protocol (RDP). Browse to https://desktop.sesync.org and login with your SESYNC username and password. Select one of the virtual machines to connect to its desktop (only machines you have permission to access are shown).

Do I have to use the cluster?

We highly recommend using the scheduled cluster for running all of your CPU-intensive or long running programs. Below is SESYNC policy for long running processes on our different types of resources:

What's the procedure for having remote participants at my meeting?

When you submit your travel planner to our travel office, please make sure to include a list of all remote participants that will be joining your meeting. You may add participants after the 8 week deadline but we require that you notify us of all participants at least one week befor ethe start of the meeting.

What resources are available in the conference rooms?

SESYNC has five conference rooms and a large breakout space equipped with HD screens, projectors, whiteboards and an array of collaborative tools. Please note, depending on the size and requests of your group, your assigned meeting room may have a different table arrangement that waht is show in the pictures below.

How do I get a SESYNC account?

Anyone who is a PI on a science team, or is part of a long running pursuit, will receive an email with instructions on completing account setup prior to their first meeting.

How do I create an RStudio project with git?

RStudio projects are folders that contain project files and a special .Rproj file. To link an RStudio project with a git repository, follow these steps:

How do I change my SESYNC password?

Point your web browser to https://pwm.sesync.org.

What is my SESYNC username?

A SESYNC username is usually your first initial followed by your last name, (i.e. “John Smith” is jsmith). Common or very long names may not follow this pattern.

What resources are available?

SESYNC has an extensive set of computing resources and expertiese available for researchers. Download a high-level overview of all services and support SESYNC offers for general information, or scan the tables below for a quick reference. Direct all questions to cyberhelp@sesync.org.

How do I contact IT or research support staff?

Email cyberhelp@sesync.org with your question or support request.

Best Practices for SESYNC Virtual Meetings

Yet another set of recommendations for how to transition to remote work!? Yes! However, this post presents a curated list of strategies specifically to help leaders of synthesis teams who are faced with the prospect of holding one or more Pursuit meetings online rather than at our center in Annapolis, Maryland. Our model thus far has emphasized the value of those in-person meetings; we even designed the physical layout to maximize interaction! However, like you, we’re now adapting. We will continue to provide online computing environments and support for your collaborations, and support new meeting formats in order to help you build or maintain team collaborations and keep forward momentum on research.

Oh, the Places You Can Get Census Population Data For!

Every 10 years, the U.S. Census Bureau conducts a nationwide survey to count the number of people in the nation, which is known as The Decennial Census. Although seemingly a straightforward concept, using these data to appropriately quantify patterns or trends1 for any given location within the country may require getting acquainted with some nuanced jargon. This post is to introduce some concepts to help you get started. e.g. considering effects of the modifiable areal unit problem ↩

Google Dataset Search: A very helpful and definitely not evil tool for finding data

Even though Google has attracted its fair share of controversy, I have to admit Google’s got to where they are because their tools are pretty good. Recently I stumbled across another of their tools I’m finding really useful: the Google Dataset Search.

Databases, huh? What are they good for?

Synthesis research involves assembling multiple data sets from different sources. Integrating those data into a format that facilitates exploration, visualization, and eventual analysis is often the most time-consuming and tedious part of the research process—however, careful attention and a little bit of strategy at early stages can pay huge dividends later on.

Using the rslurm package to run code in parallel

This blog post will walk you through a quick example of how to use the rslurm package to parallelize your code.

ggplot tricks not to forget about

Tweaking figures for presentations or publications can be a tedious process, especially when I always need a reminder on “how to use greek letters or subscripts in y-axis”, “remove legend”, and “r pch”. Here are a collection of some ggplot2 functions and arguments that I find particularly useful and want to remember.

Creating visualizations with DiagrammeR

Have you ever needed to create a visualization of a research process or statistical model that isn’t directly plotted from data? For example, a conceptual diagram, mind map, flowchart of your research process, or statistical model diagram. The R package DiagrammeR makes it much easier to create high quality figures and diagrams in situations like these.

Publishing Data Papers

Alongside sharing and publishing data sets, there are a variety of ways to publish accompanying journal articles to provide a “data description” that either includes or refers to a specific dataset. This is a way to offer narrative context beyond standard metadata, such as describing the motivation and process behind compiling the dataset being described. Additionally, this type of publication can offer formal recognition for all team members involved in creation of the dataset.

Sharing your RShiny App

RShiny and related packages have lowered the bar for making web applications in R without requring knowledge of the languages of web browsers (CSS, Javascript, HTML). This also means that sharing your app usually requires finding a platform that can run R code. Here are some (non-mutually exclusive!) options to consider for making your Shiny apps available on the web.

Adventures in Windows Dynamic Memory

SESYNC’s Windows virtual machines are setup to use dynamic memory. What this means is that your virtual machine will show different memory usage based on its current usage, however, you will still have access to the full amount we allocated to you.

Raster Change Detection Analysis with Two Images

Raster Change analysis with Two dates: Hurricane Rita

Making "dataspice" at #runconf18

As a perk of being an rOpenSci fellow, I recently got to attend the organization’s 5th ‘unconference’. This meeting brought together around 60 R users from around the world to spend a few days cooking up some new tools for the R community based on ideas discussed online leading up to the event.

Build a Shiny App to Browse MODIS Data

In preparation for our recent geospatial short course, I spent some time getting up to date on the new features in the leaflet R package. There are so many possibilities between the new add-ons in “base” leaflet, like inset mini maps and measuring tools, and even more functionality being added all the time in leaflet.extras, mapedit, and mapview.

Standardizing Non-standard Evaluation in R

Partway through her LTER Postdoc at SESYNC, ecologist Meghan Avolio ran into trouble manipulating her data on plant communities with dplyr functions. I had encouraged Meghan to modularize her scripts by writing functions for common steps in her pipeline (such as converting count data into rank-abundance curves). “You’ll love writing functions!” I said wrongly.

Writing Data Management Plans

Many funding agencies require proposals to include a section addressing plans for data management. This includes how you will handle data as it is being collected during the project, as well as plans for sharing and archiving once the project is complete. Here is a collection of resources we’ve found helpful for writing DMPs:

Images for Data Exploration in RShiny Apps

Photos, as a source of data, or to aid in the interpretation of data, can be a useful addition to RShiny applications. Here are two examples of using photo data: one that displays images from URLs, and another that uses species names to find pictures of animals.

RMarkdown

Extend your data pipeline with RMarkdown and Shiny.

Leaflet Maps

Make interactive maps in R using the leaflet package.

Data APIs

Acquire data from websites and APIs.

Model Formulas

Write formulas for regression in R and Stan.

Plots in R

Craft publication-quality graphics with ggplot2.

Spatial NetLogo

Use spatial data in NetLogo ABMs.

Spatial R Packages

Manipulate geospatial data with open source tools.

Shiny Apps

Get interactive with the Shiny R package.

Text Mine

Carve your texts into structured data.

OSGeo

Meet the open source stack underlying geospatial data.

NetLogo Scripting

Implement open agent-based models.

Relational Databases

Make your data safe, scalable and relational.

Basic R

Start learning R in RStudio.

Basic git

Learn to use git with GitHub in RStudio.

Basic Python

Start learning Python with Pandas and Scikit-learn.

Maps with R

Tour R packages that make static and interactive maps.

tidyr & dplyr

Get your data in shape with tidyr & dplyr.