Sign in
Ctrl K

Noodles

Task-based parallel programming model in Python that offers the same intuitive interface when running complex workflows on your laptop or on large computer clusters.

4
mentions
9
contributors

Cite this software

What Noodles can do for you

  • Enables scientists to execute and restart parallel workflows by readable and easily maintainable Python code
  • Helps to scale computations in Python with complex dependencies to a parallel environment
  • No need to leave the comfort of Python: Noodles is a thin layer, unobtrusively handling complexity of a concurrent environment.

Often, a computer program can be sped up by executing parts of its code in parallel (simultaneously), as opposed to synchronously (one part after another).

A simple example may be where you assign two variables, as follows

a = 2 * i

and

b = 3 * i.

Either statement is only dependent on i, but whether you assign a before b or vice versa, does not matter for how your program works. Whenever this is the case, there is potential to speed up a program, because the assignment of a and b could be done in parallel, using multiple cores on your computer's CPU. Obviously, for simple assignments like

a = 2 * i,

there is not much time to be gained, but what if a is the result of a time-consuming function, e.g.

a = very_difficult_function(i)?

And what if your program makes many calls to that function, e.g.

list_of_a = [very_difficult_function(i) for i in list_of_i]?

The potential speed-up could be tremendous.

So, parallel execution of computer programs is great for improving performance, but how do you tell the computer which parts should be executed in parallel, and which parts should be executed synchronously? How do you identify the order in which to execute each part, since the optimal order may be different from the order in which the parts appear in your program. These questions quickly become nearly impossible to answer as your program grows and changes during development. Because of this, many developers accept the slow execution of their program only because it saves them from the headaches associated with keeping track of which parts of their program depend on which other parts.

Enter Noodles.

Noodles is a Python package that can automatically construct a callgraph for a given Python program, listing exactly which parts depend on which parts. Moreover, Noodles can subsequently use the callgraph to execute code in parallel
on your local machine using multiple cores. If you so choose, you can even configure Noodles such that it will execute the code remotely, for example on a big compute node in a cluster computer.

Keywords
Programming languages
  • Jupyter Notebook 51%
  • Python 47%
  • Lua 1%
  • Shell 1%
License
  • Apache-2.0
</>Source code

Participating organisations

Natural Sciences & Engineering
Natural Sciences & Engineering
Netherlands eScience Center

Mentions

Contributors

Arnold Kuzniar
Arnold Kuzniar
Netherlands eScience Center
Ben van Werkhoven
Ben van Werkhoven
Netherlands eScience Center
Berend Weel
Berend Weel
Netherlands eScience Center
Felipe Zapata
Felipe Zapata
Netherlands eScience Center
HS
Hanno Spreeuw
Netherlands eScience Center
JB
Joris Borgdorff
Netherlands eScience Center
Lars Ridder
Lars Ridder
Netherlands eScience Center
Vincent van Hees
Vincent van Hees
Netherlands eScience Center

Related projects

Fast open source simulator of low-energy scattering of charged particles in matter

Transferring code to the larger community

Updated 12 months ago
Finished

eStep

Developing an eScience technology platform

Updated 13 months ago
Finished

Computational Chemistry Made Easy

Bringing concepts from distributed computing and bioinformatics to the field of computational...

Updated 13 months ago
Finished

Related software

PyXenon

PY

Python wrapper for the Xenon programming interface to various compute and storage resources.

Updated 21 months ago
3

Xenon

XE

If you are using remote machines to do your computations, and don’t feel like learning and implementing many different APIs, Xenon is the tool for you.

Updated 6 months ago
13 11

Xenon gRPC server

XE

Run applications remotely via a gRPC interface.

Updated 6 months ago
4 3