Thomas Rüegg

Details

• Designed and developed a small web app to create and vote on polls with a swipe interface
• Uses NextJs, React, Typescript and Tailwind CSS with Postgres as database

Details

• Designed and developed a personal website using Next.js, React and Tailwind CSS

Details

• Developed a Telegram bot which can tell you which sauna is open currently in Switzerland.
• The bot was capable of telling you exactly if a sauna was open, how much it costs, took into account gender specific restrictions and how long it would take to get there via the SBB API

Abstract

This thesis explores the development of efficient multitask neural networks towards enabling complex visual tasks on resource-constrained devices.

Abstract

Intelligent edge vision tasks encounter the critical challenge of ensuring power and latency efficiency due to the typically heavy computational load they impose on edge platforms. This work leverages one of the first “Artificial Intelligence (AI) in sensor” vision platforms, IMX500 by Sony, to achieve ultra- fast and ultra-low-power end-to-end edge vision applications. We evaluate the IMX500 and compare it to other edge platforms, such as the Google Coral Dev Micro and Sony Spresense, by exploring gaze estimation as a case study. We propose TinyTracker, a highly efficient, fully quantized model for 2D gaze estimation designed to maximize the performance of the edge vision systems considered in this study. TinyTracker achieves a 41x size reduction (~ 600Kb) compared to iTracker [1] without significant loss in gaze estimation accuracy (maximum of 0.16 cm when fully quantized). TinyTracker's deployment on the Sony IMX500 vision sensor results in end-to-end latency of around 19ms. The camera takes around 17.9ms to read, process and transmit the pixels to the accelerator. The inference time of the network is 0.86ms with an additional 0.24 ms for retrieving the results from the sensor. The overall energy consumption of the end-to-end system is 4.9 mJ, including 0.06 mJ for inference. The end-to-end study shows that IMX500 is 1.7x faster than Coral Micro (19ms vs 34.4ms) and 7x more power efficient (4.9mJ VS 34.2mJ).

Abstract

Detection and description of keypoints in images is a fundamental component of a wide range of tasks such as Simultaneous Localization And Mapping (SLAM), image alignment and structure from motion (SfM). Efficient computation of these features is crucial for real-time applications and has been addressed by multiple handcrafted algorithms and, recently, by deep neural network-based detectors. Learned detectors achieve high detection performance, but pose high computational requirements, making them slow and impractical for low-power resource constraint platforms. This paper presents a quantized neural keypoint detector and descriptor optimized for edge devices exploiting two recent AI platforms such as MAX78000 by Analog Devices and the Coral AI USB accelerator from Google. To accommodate the diverse constraints and requirements of various applications, we propose and evaluate two model architectures (KP2DtinySmall and KP2DtinyFast) and deploy them on the aforementioned platforms using full 8-bit integer quantization. Furthermore, we extensively evaluate these models in terms of power, latency and accuracy, reporting results on three image sizes (88x88, 320x240 and 640x480), evaluating both quantized and non-quantized models. Fully quantized, KP2DtinySmall reduces network size by a factor of 54x while improving homographic estimation accuracy on 88x88 images on the most stringent threshold (Correctness d1) by 32.4% (0.550) and on 320x240 images by 10.7% (0.648) compared to the KeypointNet architecture by Yang You et. al. This result is achieved by designing a new network with low power platforms in mind, particularly addressing the lower resolution by increasing the density of detectable features. Deployed on the MAX78000 MCU, inference of low-resolution images is run at 59 FPS, consuming 1.1 mJ per image. On the Coral usb accelerator, KP2DtinyFast runs inference on low-resolution images at 527 FPS consuming 3.1 mJ, on high resolution it achieves 70 FPS at 19.9 mJ per inference.

Abstract

This thesis explores unsupervised learning techniques for feature point detection and description in multi-beam imaging sonar data, enhancing underwater object recognition and mapping. (In collaboration with Tethys Robotics)

Abstract

This project implements a semantic instance fusion algorithm in Python, combining 3D reconstruction with semantic segmentation for improved scene understanding.