📑 Background

Neuromorphic technology (NT) continues to attract attention as an alternative to mainstream AI technology, such as GPU-accelerated deep learning. Some promising aspects of NT include energy and data efficiency, low latency, ability to handle spatiotemporal data and (specifically for hardware platforms) a small form factors. These traits make NT promising for embedded and edge applications. Recent advances in both algorithms and hardware are making the case for adopting neuromorphic technology even more appealing and feasible.

🔀 Neuromorphic Intermediate Representations

As the number of different hardware platforms and software simulation tools grows, the question about interoperability is gradually moving into the spotlight. The Neuromorphic Intermediate Representations (NIR) framework aims to address this point by providing an interoperability layer that can facilitate the implementation of neuromorphic circuits on different types of hardware.

The NIR framework is the analogue of ONNX in the neuromorphic field. It has gradually gained traction since its inception and initial implementation, and it currently supports multiple software and hardware platforms:

👉 Table source

The NIR framework aligns well with the goals of Action Line 2 of the Neuromorphic Computing NL Consortium, having the potential to enable research and applications to benefit from enhanced interoperability across what is now a heterogeneous and fragmented hardware and software landscape.

👁️ Neuromorphic vision

So far, the development of NIR has focused on computational primitives suitable for representing the behaviour of spiking artificial neurons. Considering the diversity and complexity of sensory systems in the biological world, it is only natural to extend NIR with biosensor primitives. As a first step, it would be beneficial to add support for neuromorphic vision since the physiological and algorithmic aspects of visual perception have been the subject of extensive studies for many a century.

Currently, the most well-established neuromorphic vision technology is the event-based sensor, more commonly known as an event camera. It deviates significantly from traditional frame-based CMOS cameras, boasting low latency, high dynamic range and data sparsity. The event camera is the culmination of decades of research and prototyping that has integrated insight from neuroscience accumulated over even longer periods of time.

Adding support for visual primitives to NIR would reduce the time and effort associated with designing and demonstrating more sophisticated vision sensor prototypes.

🎯 Project Goals

The main goal of this project is to augment NIR with new features targeting computational primitives for neuromorphic vision. A core part of this approach is adding support for NIR to the Pyrception simulation framework. As a high-level workflow, Pyrception can be used for designing high-level visual filters, including sparse retinomorphic cell arrangements and complex receptive fields with custom shapes, sizes and orientations. Once a filter is designed and verified to work in simulation, it can be exported to a NIR graph that can be subsequently implemented in hardware by using an FPGA.

The long-term goal of the project is to model visual circuitry for specific applications such as optical flow estimation, egomotion subtraction, looming detection and object tracking. Filters and circuits for such applications can then be published to the SynFire repository, greatly reducing the effort associated with development and deployment.