Cyberhelp

If you have a program that runs only on Windows systems, and you would like to run the program in parallel, it is not possible to use SESYNC’s Slurm cluster. However, it is possible to take advantage of SESYNC’s Winanalytics virtual machine, which has multiple cores and much more available RAM than the typical laptop. You will need to write a little bit of code in PowerShell, which is Windows’ shell scripting languge and is fairly similar to Bash scripting. Here is a quick walkthrough of how to do this.

SESYNC provides a high-performance computing cluster for memory-intensive and time-intensive computing tasks. (FAQ: What is the SESYNC cluster?) You can connect to the cluster through our ssh gateway service running at ssh.sesync.org or by submitting jobs through RStudio. The workflow for using a cluster is a little bit different from a typical run in R or python. In addition to your processing code, you must give the cluster a list of execution instructions and a description of the resources your analysis will require. Any output from your script will be written out to a file called slurm-[jobID].out and errors go to slurm-[jobID].err.

To support existing data analysis pipelines that use the Stata software, SESYNC has purchased a Stata license and created a dedicated virtual machine for remote use by affiliated researchers. This quick start guide explains the essential steps for evaluating Stata commands over SSH or from SESYNC’s RStudio server.

Nextcloud

Navigate to lists.sesync.org in a Web Browser

This Quick Start guide will walk you through establishing a connection to a database on SESYNC’s server. Access to your pursuit’s relational database management system (RDBMS) requires communication between a server application (PostgreSQL or MySQL) and a client application (RStudio, Jupyter, psql, etc.).

SESYNC provides access to remote JupyterLab sessions via a web browser. The Jupyter Project provides an environment for Python development, and SESYNC’s Jupyter Server adds direct connections to resources like shared file storage, databases, GitLab, and a compute cluster.

Choosing to publish your data products in a long-term repository can:

SESYNC researchers and staff can provide a link for external collaborators to upload (donwload) files to (from) any research data directory they can access. We recommend this mechanism for receiving datasets from external collaborators: create a new folder and turn it into a public “file drop”, as described below.

SESYNC provides access to a remote RStudio session, via a web browser, in order to work in R while directly connected to other SESYNC resources (file storage, databases, the cluster, etc).

When you submit your travel planner to our travel office, please make sure to include a list of all remote participants that will be joining your meeting. You may add participants after the 8 week deadline, but we require that you notify us of all participants at least one week before the start of your meeting.

To publish a R shiny application on the SESYNC server, your files will need to be copied from your working directory to the shiny-apps-data shared folder (/nfs/shiny-apps-data on RStudio Server). Please contact SESYNC IT staff if you would like to host an app on SESYNC’s Shiny Server.

SESYNC offers private git hosting through our GitLab server. When you connect to our GitLab Community Edition (CE) instance using your SESYNC username and password, you’ll see a dashboard of recent activity on projects that you are part of. If this is your first time connecting, it may be a little quiet.

You can upload and download data from your research data directory using an SFTP client. We recommend Cyberduck or WinSCP

SESYNC provides a large, shared file store to host data for all projects. Project participants have access to the research data directory for their project from our compute servers, a web portal, a desktop application for syncing, and SSH.

SESYNC has installed the eBeam whitebaord capture software on all of our conference room PC’s and laptops.

Use of video conference platforms has exploded now that we are all working from home.

There are a few different ways to run a job on SESYNC’s Slurm compute cluster, but all of them ultimately run a command called sbatch to submit the job to the cluster. The sbatch program is part of the Slurm software package and has a lot of different options. These include a maximum length of time your jobs can run, how much memory you are requesting, whether you want to be notified by email when your job finishes running, etc. It’s possible to run a Slurm job without setting any of the options and going with all defaults, but there are times when you might want to customize the options.

There’s no user-friendly way to do this. You have to use git in the command line.

SESYNC’s Slurm compute cluster allows users to run big memory- and processor-intensive jobs. Many users don’t know that you can access the memory and processing power of the cluster interactively, typing commands directly into the command line or into an R or Python session. This FAQ briefly describes how to start an interactive job on the Slurm cluster.

Many jobs on the Slurm compute cluster generate lots of big files that require large amounts of memory to be stored but are only needed temporarily. There are two different ways to easily store large temporary files created by cluster jobs: temporary storage on a specific node (/tmp/) and scratch space accessible from all nodes (/nfs/scratch/).

TL;DR: Try to have a general idea of your data storage needs, and discuss it with the data science team if you are concerned, but do not be too worried unless you are going well over 1 terabyte.

There are several resources available for collaborative writing depending on which platform you prefer to work. These are the resources SESYNC groups have successfully used in the past.

To access and see your data directory from the side data console, you need to first set a symlink to your research directory. To do this, open a terminal notebook in Jupyter lab. The command to set the symlink is:

Please see the cyberinfrastructre “Process & Policies” page on SESYNC’s main website.

SESYNC’s data storage and computational resources are available to pursuit participants for approximately one year after the final meeting.

It is, or at least will be as soon as you need it! Any research data directory you have access to will be mounted to the filesystem at “/nfs” when you access it. If you have not touched any of the files in there for a while, it may have un-mounted and appear to be missing. So if you don’t see your “*-data” folder under “/nfs”, just navigate directly to the folder and it will instantly mount. For example, if your research data directory is “cooltrees-data”, then enter the full path as “/nfs/cooltrees-data” in the file browser or from the command line.

If you are a remote participant, see the below instructions the download the Skype for Business pugin and join your meeting.

SESYNC Windows client virtual machines are setup to use dynamic memory. What this means is that your virtual machine will show a different amount of memory available based on its current usage. You still have access to the full amount of memory allocated if needed. The virtual machine will grab more memory from the hypervisor when needed automatically.

Due to some quirks on our storage system your git repo may show that all of your files have modifications. If you perform a ‘git diff’ you will see a list that looks like:

A virtual machine is a Windows or Linux machine that runs on and shares computing resources with a physical machine known as a hypervisor. Virtual machines allow the deployment of multiple machines or services on one or several hypervisors to better utilize computing resources (CPU cores, memory, etc…)

SESYNC has the ability to deploy custom Windows and Linux Virtual Machines for use by groups. If there is a software or service needed that is not provided by our shared infrastructure, we can deploy a virtual machine to meet your needs.

SESYNC can work with your group to obtain software that would be beneficial to your research while at SESYNC. All software purchased by SESYNC must be installed on SESYNC-owned equipment. We can provide virtual machines to use by your group members to access your purchased software for use.

When are the server maintenance windows?

We’re sensitive to the fact that your jobs may need to run over our maintenance window and will take a reasonable effort to ensure they aren’t disrupted. In order to ensure as minimal disruption as possible, these are the steps that we take:

SESYNC uses the sympa mailing list server to manage mailing lists. You can manage your list by going to lists.sesync.org and entering your e-mail address and SESYNC password in the upper right.

Shiny applications hosted by SESYNC are publically available at the URL http://shiny.sesync.org/apps/<APPNAME>, where <APPNAME> is unique for each app. To get started on publishing an app, read the quick start guide.

SESYNC Linux resources are deployed on a private network at SESYNC and are accessed via our ssh gateway at ssh.sesync.org. These resources include RStudio, Jupyter lab, and our compute cluster. Please DO NOT run your computational processing on the ssh gateway, it has limited memory and processing power. Instead, use the ssh gateway to submit jobs to SESYNC’s compute cluster or to connect to your virtual machine.

SESYNC offers the ability for your group to schedule conference calls outside of your onsite meetings. These calls can either be dial-in only, or audio-video. To schedule a call, please contact SESYNC it staff cyberhelp@sesync.org at least 7 days in advance of your call and we will configure a one time or standing meeting for you. Please let your participants know to keep a lookout for a meeting invitation from SESYNC’s IT staff. After you schedule your call, the following will happen:

Navigate to https://files.sesync.org and log in with your SESYNC username and password. The folders listed under “External storages” are each a shared research data directory accessble to participants in the corresponding project.

For new groups, we generally follow the timeline below:

If you already have projects on GitHub that you are working on, we prefer that you continue to use GitHub due to its open nature. We’ll gladly push and pull code from your public repository. We provide GitLab locally for projects that are just starting up, have sensitive data, or are not quite mature enough to be pushed out into the world.

The three are often a source of confusion.

Yes! You can push a local git repository to any new remote resource. Please note that only your source code will move. However, the additional features you use (e.g. wiki, issues, etc.) will need to be manually copied.

SESYNC’s computational cluster (see quickstart page) enables users to run medium-to-large scale analyses by distributing multiple, independent tasks across many computers. This setup is ideal for tasks that require applying the same algorithm or a parameter set over independent units in a large data set.

SESYNC provides remote access to all desktop resources through a browser based Remote Desktop Protocol (RDP). Browse to https://desktop.sesync.org and login with your SESYNC username and password. Select one of the virtual machines to connect to its desktop (only machines you have permission to access are shown).

We highly recommend using the scheduled cluster for running all of your CPU-intensive or long running programs. Below is SESYNC policy for long running processes on our different types of resources:

When you submit your travel planner to our travel office, please make sure to include a list of all remote participants that will be joining your meeting. You may add participants after the 8 week deadline but we require that you notify us of all participants at least one week befor ethe start of the meeting.

SESYNC has five conference rooms and a large breakout space equipped with HD screens, projectors, whiteboards and an array of collaborative tools. Please note, depending on the size and requests of your group, your assigned meeting room may have a different table arrangement that waht is show in the pictures below.

Anyone who is a PI on a science team, or is part of a long running pursuit, will receive an email with instructions on completing account setup prior to their first meeting.

RStudio projects are folders that contain project files and a special .Rproj file. To link an RStudio project with a git repository, follow these steps:

Point your web browser to https://pwm.sesync.org.

A SESYNC username is usually your first initial followed by your last name, (i.e. “John Smith” is jsmith). Common or very long names may not follow this pattern.

SESYNC has an extensive set of computing resources and expertiese available for researchers. Download a high-level overview of all services and support SESYNC offers for general information, or scan the tables below for a quick reference. Direct all questions to cyberhelp@sesync.org.

Email cyberhelp@sesync.org with your question or support request.

Every 10 years, the U.S. Census Bureau conducts a nationwide survey to count the number of people in the nation, which is known as The Decennial Census. Although seemingly a straightforward concept, using these data to appropriately quantify patterns or trends1 for any given location within the country may require getting acquainted with some nuanced jargon. This post is to introduce some concepts to help you get started. e.g. considering effects of the modifiable areal unit problem ↩

Even though Google has attracted its fair share of controversy, I have to admit Google’s got to where they are because their tools are pretty good. Recently I stumbled across another of their tools I’m finding really useful: the Google Dataset Search.

What’s a merge conflict?

Synthesis research involves assembling multiple data sets from different sources. Integrating those data into a format that facilitates exploration, visualization, and eventual analysis is often the most time-consuming and tedious part of the research process—however, careful attention and a little bit of strategy at early stages can pay huge dividends later on.

This blog post will walk you through a quick example of how to use the rslurm package to parallelize your code.

Tweaking figures for presentations or publications can be a tedious process, especially when I always need a reminder on “how to use greek letters or subscripts in y-axis”, “remove legend”, and “r pch”. Here are a collection of some ggplot2 functions and arguments that I find particularly useful and want to remember.

Have you ever needed to create a visualization of a research process or statistical model that isn’t directly plotted from data? For example, a conceptual diagram, mind map, flowchart of your research process, or statistical model diagram. The R package DiagrammeR makes it much easier to create high quality figures and diagrams in situations like these.

Alongside sharing and publishing data sets, there are a variety of ways to publish accompanying journal articles to provide a “data description” that either includes or refers to a specific dataset. This is a way to offer narrative context beyond standard metadata, such as describing the motivation and process behind compiling the dataset being described. Additionally, this type of publication can offer formal recognition for all team members involved in creation of the dataset.

RShiny and related packages have lowered the bar for making web applications in R without requring knowledge of the languages of web browsers (CSS, Javascript, HTML). This also means that sharing your app usually requires finding a platform that can run R code. Here are some (non-mutually exclusive!) options to consider for making your Shiny apps available on the web.

SESYNC’s Windows virtual machines are setup to use dynamic memory. What this means is that your virtual machine will show different memory usage based on its current usage, however, you will still have access to the full amount we allocated to you.

Raster Change analysis with Two dates: Hurricane Rita

As a perk of being an rOpenSci fellow, I recently got to attend the organization’s 5th ‘unconference’. This meeting brought together around 60 R users from around the world to spend a few days cooking up some new tools for the R community based on ideas discussed online leading up to the event.

In preparation for our recent geospatial short course, I spent some time getting up to date on the new features in the leaflet R package. There are so many possibilities between the new add-ons in “base” leaflet, like inset mini maps and measuring tools, and even more functionality being added all the time in leaflet.extras, mapedit, and mapview.

Partway through her LTER Postdoc at SESYNC, ecologist Meghan Avolio ran into trouble manipulating her data on plant communities with dplyr functions. I had encouraged Meghan to modularize her scripts by writing functions for common steps in her pipeline (such as converting count data into rank-abundance curves). “You’ll love writing functions!” I said wrongly.

Many funding agencies require proposals to include a section addressing plans for data management. This includes how you will handle data as it is being collected during the project, as well as plans for sharing and archiving once the project is complete. Here is a collection of resources we’ve found helpful for writing DMPs:

Photos, as a source of data, or to aid in the interpretation of data, can be a useful addition to RShiny applications. Here are two examples of using photo data: one that displays images from URLs, and another that uses species names to find pictures of animals.

Extend your data pipeline with RMarkdown and Shiny.

Make interactive maps in R using the leaflet package.

Acquire data from websites and APIs.

Write formulas for regression in R and Stan.

Craft publication-quality graphics with ggplot2.

Use spatial data in NetLogo ABMs.

Manipulate geospatial data with open source tools.

Get interactive with the Shiny R package.

Carve your texts into structured data.

Meet the open source stack underlying geospatial data.

Implement open agent-based models.

Make your data safe, scalable and relational.

Start learning R in RStudio.

Learn to use git with GitHub in RStudio.

Start learning Python with Pandas and Scikit-learn.

Tour R packages that make static and interactive maps.

Get your data in shape with tidyr & dplyr.