SoftwareApplication
identifier: 10.5281/zenodo.996294
name: Emma
description: Emma is a project to create a platform for development of application for Spark and DockerSwarm clusters.
description: * It is designed for users deploying Spark and DockerSwarm clusters in a cloud infra-structure. * It helps the user to prepare cloud virtual machines * The provision of machines is done with Ansible, an automation tool for IT infra-structure. * It provides command line access to the users to install the required libraries and systems, configure them, start/stop services, add new modules for Jupyter notebooks, and even update the firewall Emma is an open-source project to create a platform for development of applications for Spark and DockerSwarm clusters. The platform runs on an infra-structure composed by virtual machines that must be reachable by SSH. The machines are either cloud virtual machines or Vagrant machines. The latter tool allows the platform to be simulated on a local machine, i.e. in a local development environment. Once the machines are prepared, the servers are provisioned using Ansible, an automation tool for IT infra-structure. Ansible playbooks are used to create a storage layer, processing layer, and JupyterHub services. The storage layer offers two flavors of storage, file-base by GlusterFS and Hadoop Distributed File System (HDFS), and object-based using Minio. The processing layer has a Apache Spark cluster and a Docker Swarm sharing the storage instances. With Ansible we are able to deploy a platform with the same features at different locations, such as local cluster, national infra-structure, or even a commercial cloud provider. Such a feature allows us to have tool-provenance for easily repeatability of experiments between scientists.
code repository: https://github.com/nlesc-sherlock/emma
author: https://orcid.org/0000-0003-2225-1428
author: https://orcid.org/0000-0002-5821-2060
author: https://orcid.org/0000-0002-0948-1176
author: https://orcid.org/0000-0001-9795-7981
keywords: Big data
programming language: Python
programming language: R
programming language: Scala
programming language: Shell
license:
  creative work: creativeWork
    name: Apache-2.0

https://orcid.org/0000-0003-2225-1428
type: Person
given name: Romulo
family name: Gonçalves
name: Romulo Gonçalves
affiliation: Organization
  name: Netherlands eScience Center

https://orcid.org/0000-0002-5821-2060
type: Person
given name: Stefan
family name: Verhoeven
name: Stefan Verhoeven
affiliation: Organization
  name: Netherlands eScience Center

https://orcid.org/0000-0002-0948-1176
type: Person
given name: Jisk
family name: Attema
name: Jisk Attema
affiliation: Organization
  name: Netherlands eScience Center

https://orcid.org/0000-0001-9795-7981
type: Person
given name: Niels
family name: Drost
name: Niels Drost
affiliation: Organization
  name: Netherlands eScience Center