Rise of AI Conference 2019 · Berlin · Unicorn Stage

Using Sound and AI for Low-Cost 3D Environment Perception

Tobias Bahnemann · Managing Director, Toposens

At Rise of AI 2019 in Berlin, Tobias Bahnemann, co-founder and managing director of Toposens, presents a 3D ultrasound sensor that uses sound and AI to give robots and vehicles low-cost close-range environment perception. Comparing the idea to Batman's citywide sonar in The Dark Knight, he explains that Toposens sends out high-frequency sound, captures reflections with three microphones, and processes them on a small chip, the SoundVision 1, into a 3D point cloud, without needing light and while generating only 3 to 5 megabytes of data per hour, small enough for on-device AI processing. He argues the technology fills a gap left by camera and lidar systems, which struggle at close range, in poor lighting, with glass, or at prices that keep them out of many products, citing failures such as unreliable autonomous parking and undetected curbs. He describes four prerequisites for training reliable neural nets on this data (reliable, precise, dense and large datasets) and Toposens' staged roadmap from detecting static objects to dynamic ones and eventually predictive understanding, alongside partnerships with Nvidia and Infineon, named customers, and an ongoing funding round.

Key takeaways

  • Toposens' 3D ultrasound sensor works independent of lighting conditions and costs less than comparable camera or lidar systems.
  • The sensor produces only 3 to 5 megabytes of data per hour, small enough to run AI processing on the edge device.
  • Bahnemann identifies four prerequisites for training a neural net on this sensor data: reliability, precision, density and large datasets.
  • Toposens positions its five-meter-range sensor as complementary to lidar and cameras, not a replacement, especially for close-range maneuvering.
  • The company had raised about two million dollars from Silicon Valley and German investors and was starting a new funding round at the time of the talk.

Detailed notes

A bat-inspired, low-cost 3D sensor

Bahnemann introduces Toposens by comparing its idea to Batman's fictional citywide sonar system in The Dark Knight, calling the company's device the real-world equivalent: a sensor that perceives its surroundings through sound the way bats do. The company's Initial Sense product, about to launch commercially within two months of the talk, sends out high-frequency sound, receives the reflected signals, and processes them into a three-dimensional point cloud. He highlights that the system works regardless of lighting conditions, is the cheapest 3D sensing technology on the market because it needs only a small processing chip, is optimized for short range up to five meters, and generates only 3 to 5 megabytes of data per hour of operation, small enough that AI processing can run on the edge device without uploading data.

Why close-range perception matters

He argues that autonomous systems in robotics, passenger cars and commercial vehicles have so far relied mainly on optical 3D cameras and lidar, both of which struggle at short distances or are too expensive to equip broadly, contributing to deaths, injuries and physical damage. As examples, he cites luxury cars whose autonomous parking functions work only about half the time due to poor sensor data, and demonstrates the sensor mapping a sofa in a home environment for vacuum-cleaner-robot applications, noting that unlike optical systems it detects glass well.

How the sensor works

Bahnemann explains that on the hardware side the system needs only a transducer, three microphones and the company's SoundVision 1 chip, plus a circuit board and housing; on the software side, the chip calculates the time of flight of each sound reflection and runs 3D processing to build the point cloud. He describes the current processing as lower-level AI focused on echo detection and precise timing analysis, plus a new cooperation with Infineon on transducer components and an existing partnership with Nvidia on neural nets for autonomous driving.

From reliable data to real intelligence

He lists prerequisites the team identified over the past one to two years for training a proper neural net on this data: reliable data with low discrepancy between sensors down to millimeter precision, high point density, and, once those are met, large datasets. He places Toposens between the first two of four development stages: after establishing reliable sensor data and pattern detection, the next step is recognizing static objects, followed by recognizing moving, dynamic objects, and finally what he calls real intelligence, where the system predicts what a detected person or object is about to do.

Roadmap, customers and funding

Bahnemann says Toposens is releasing its first commercial product, aimed mainly at robotics companies, in July, about two months after the talk, followed by an autonomous-driving development kit combining several sensors and, by year end, a first sample toward a commercialized automotive sensor. He names Nvidia and Infineon as partners and says the three co-founders split roles between the technical inventors of the sensing system and himself handling business and customers. The company has raised around two million dollars to date from Silicon Valley and German investors and is starting a new funding round.

Q&A highlights

Asked what role machine learning actually plays, Bahnemann concedes the sensor itself currently uses classic signal processing rather than large trained neural nets, and that the company is only now starting to train models after realizing over the past six to nine months that the data required far more preparation than expected; he positions Toposens as an enabling technology for AI rather than an AI company itself. Asked where the sensor fits in autonomous driving given its five-meter range, he calls it complementary within a full sensor stack alongside lidar and cameras, most relevant for low-speed maneuvering and parking. Asked about Tesla, he says Tesla uses twelve ultrasonic sensors and, without disclosing NDA-covered details, that a company like Tesla is very interested in the level of data Toposens can provide. Asked about broader sound applications such as detecting sirens, he says the system is largely hardware-agnostic, and confirms that standard optical-style image recognition does not work directly on the sound-derived point cloud data; new types of neural nets tailored to its patterns still need to be developed.

Quotes

For the first time ever we can understand our surroundings like bats can perceive their environment through ultrasound.

Tobias Bahnemann

Questions answered in this talk

How does Toposens' 3D ultrasound sensor work?

According to Tobias Bahnemann at Rise of AI 2019, the sensor emits high-frequency sound pulses, captures the reflections with three microphones, and calculates time of flight on its SoundVision 1 chip to build a three-dimensional point cloud, similar to how bats use echolocation.

Why is ultrasound sensing useful for autonomous driving and robotics?

According to Tobias Bahnemann, ultrasound sensing works regardless of lighting, is far cheaper than camera or lidar systems, and performs well at the close range, such as curbs and parking maneuvers, where existing optical systems often fail.

What is needed to train a reliable neural net on 3D ultrasound sensor data?

According to Tobias Bahnemann, Toposens found four prerequisites: the sensor data must be reliable and consistent across units, precise down to the millimeter, dense enough per point cloud, and available in large quantities before a proper neural net can be trained.

Mentioned

3D ultrasound sensing, SoundVision 1, time of flight, edge AI, point cloud, echolocation

Keep exploring

More from Rise of AI Conference 2019

More on Industry & Robotics

Transcript

Auto-generated transcript, not edited. · 3,725 words

Show the full transcript

hello everybody yeah my name is Tobias I'm one of the cofounders of topple sends and yeah I want to explain to you how tapas ends uses yes sound and AI to come up with a very low-cost 3d perception system to understand our environment better and this introduction by a spa was pretty good because when thinking about me explaining what we're doing I I came up with these with these clips I hope the sound is on so maybe you know the movie The Dark Knight yeah so the Dark Knight in this scene Batman because he's very good sonar concept and applied it to every phone in the city but half the city feeling you saw now you can image all of that so what he was saying so what what did Batman do so he implemented a 3d sonar system inside every smartphone and through everybody has a smart phone in his pocket and through emitting sound he could see all the surroundings what this comes up to the scene quality isn't that good but I'm

still have an idea what's happening okay you got veal and alpha I'm Leone beta it's like classical he's easy to see everything team farewell team so you can see the waves and being emitted hopper calm down so why am i showing this so it's oh yeah nice introduction but the most important thing is okay if you've been seeing this movie okay this is totally unrealistic that this Batman this can never happen we have to develop this so that's that's the the trick so what I'm holding here in my hand is maybe not the smartphone technology so we're not at that level yet but in general we have exactly created what you can see here in the movie so we've invented the bat sensor so for the first time ever we can understand our surroundings like bats can perceive their environment through ultrasound oh no something a bit so here on the left you can see our initial sense we are just releasing to market in about two months and our technology is mainly based on

software algorithms and artificial intelligence so because what is basically happening maybe you know how bats perceive the environment and that is very similar to what our system is doing we are sending out high frequency sound we're receiving signals we process the sound and then we get a 3d data so every one of our systems generates a three dimensional point cloud as I think to talks ago we have seen you can do with a camera or lidar system the awesome thing about our technology is that we don't care about lighting condition it's lighting conditions at all it is a very robust system next big topic is it's the cheapest 3d sensing technology worldwide because we only need a very small processing chip to do all this 3d processing that's due to ya the special software and technology we worked on now for over eight years in total what is another factor the system is not to the long range a system is very good at short distance sensing so up to five meters since we're using

sound you cannot like look for 100 meters but you can sense up to 5 meters which gives you a lot of applications in the Meishan space and then yeah another important topic especially on the lower right hand side is our system only generates a super small amount of data so when we let the system run for about an hour we're in an area of 3 to 5 megabyte in data in 3d data our system is generating so now think of what our term yeah what artificial intelligence algorithms can do with this data you can do AI processing on the edge because the data amount is so small you don't have to upload it anywhere so where are we going with this technology Automation is happening everywhere I think that is pretty clear we have already talked about it now and seen some topics if you talk about the robotic space you can look at consumer robots you can talk about industrial robots and as well as in the in the automotive space so passenger cars

autonomous driving that's what I was asking a question a bit earlier but also commercial vehicles are automated every autonomous system needs to perceive its surroundings so 3d sensors and able these cars to drive and move and interact with the surroundings autonomously up to now the dominant systems which have been doing this task were mainly 3d cameras and 3d lidar systems so both of them optical based so sending out light or receiving light and then processing it and this is exactly what yeah we are changing now our system is using sound so totally different approach why is this important so why do we need this kind of technology out there on the market so these are some of the cases where a short distance sensing is failing so the existing systems don't work properly why is this the case the environm environmentally conditions are problematic the close distance 3d data isn't good yeah the systems are just too expensive to equip your object your autonomous system with a sensor yeah so several things lead to these

kinds of failures and they cause yeah deaths injuries a lot of physical damages so close distance sensing for autonomous technologies it's just really a gap that it hasn't been solved perfectly yeah and especially especially price is also from very expensive optical systems is keeping all these technologies from using good close to since we sensing and sure that is exactly what we solving with our 3d ultrasound sensing system so maybe here I want to show you give you an idea how is our system implemented into into a driving vehicle so for maybe you know your Park distance control sensors which you already have in the car yeah so they give you one dimensional ultrasound so only the distance with our processing we are able to generate 3d data yeah so we're send out sound we get the sound reflections back we process the data and we get three dimensional point clouds as you get with other systems so if a car is driving equipped with our seven sensors all the sensors fire and send out sound

we get the reflections and we generate a 3d point cloud of the surroundings this is still a simulation but in a second you can see like real life data mm-hmm so this is what it looks like when we are driving out on the street yeah so this is close to since up to 5 meters 3d data it's a bit difficult to interpret for the human eye the data is color coded so on the lower level you have blue then you go into green orange and red and what we are able to see what is very difficult for other systems is for example curbs yeah so um small things on the ground which is especially important for autonomous driving systems in the close distance maneuvering and parking space maybe you have already tested one of the autonomous parking functionalities and one of the more luxury type cars many of them don't work only working 50 percent yeah or yeah say no there's an error I cannot I cannot park that is mainly due to bad sensor

data yeah so and that is exactly what what we are providing maybe to jump one step into the robotic space I talked about vacuum cleaner robots or I have shown them on another image so this is an idea of how our system is perceiving the closest and surroundings in your home yeah so you can see the 3d point clouds of the sofa and yeah and compared to other systems we are very good at detecting glass for example yeah so that is also another thing that is difficult with optical systems so how does this system actually work yeah so I'm talking about 3d sensors but what's what's behind this thing here I'm holding your and so initially I talked about the bat echolocation our system is working very similar to it so I have a small speaker here we send out ultrasonic pulses of sound all the surrounding objects reflect the sound so in typical echo we have microphones in here and those microphones yeah listen to the reflections and then inside our processor all the

processing is happening yeah so we calculate the time of flight so how long does it take for the sound to travel from our system to the object and back and then our algorithms do all the 3d processing that we get 3d 3d data out of the small system so in the end on the hardware side our system is very simple you need our chip recalled the sound vision 1 you need three microphones yeah we just started a cooperation with Infineon because they also see the relevance okay now we can not only listen to our surroundings with microphones we can start seeing our surroundings and the major component company for transducers for senders so those are the major components one transducer through microphones and the chip sure a bit of PCB and housing and so on but that's it and you have a 3d sensor the magic is happening inside inside the sound vision one so now what are the software blocks where's lower and higher artificial intelligence happening inside our system so up to now

most of the data processing yeah is a lower level of artificial intelligence because we are yeah looking at the data that our system is receiving and finding different things so one thing is the echo location yeah so once we process the sound the sound data we need to find where are the echos because you have a noise noise level so there's lots of noise in in in such an audio signal you need to probe what what is a real echo what is a real echo I'm and I'm seeing it that relates to an object otherwise you have many false positives the next point is the timing analysis from this audio signal you need to super super precisely find out what the timing is yeah so you need very good analytics and processing to in the billionth of a second areas on the nanosecond area that we can process 3d data and then short there's all the 3d processing that we generate this 3d point cloud and now this is the next level where it becomes

really interesting so the 3d data analysis once we generate all this 3d data what can we do with it I mean that's where how we start understanding our surroundings if since we process the the 3d 3d data we are we are seeing and I want to quickly explain to you what has to happen to get we are highly reliable yeah deep learning machine vision data out of this to understand your surroundings that is a topic we are currently working on so we don't have the perfect deep learning neural net already installed and these are four very important prerequisites which we found out over the last one to two years which you have to have before you can really train a proper neural net so one thing is you need very reliable data so every sensor needs to give you the same data yeah if you have a very high discrepancy between the different sensor systems it becomes very difficult in the very precise state area so you need to be in the very low centimeter

if that even millimeter area per point you're receiving so you need to have a very high performance then you need a high density you have maybe seen the video and I've seen okay we have rather sparse point cloud that is more difficult to train a neural net on that and then if you have a very dense point cloud which is easier than to do with optical systems and then for sure if you have met all the other three topics you need large datasets generating more and more data the easier it becomes to train unit so what are the steps for us we were working on and have ahead of us to get to real intelligence what we think about when you think about artificial intelligence typically so we are like in the directly in the middle between step one and two right now so we have reliable sensor data now and we are able to detect patterns because objects yeah give us similar patterns and then on the next level we want to yeah start

understanding different objects in a static environment you place your object there you scan it I don't know thousand times a hundred times and replace it with a similar object with maybe different bit of a different shape and put in a static environment that's like the next tasks now when you think about all our vehicles we want to apply these sensors the next level is dynamic objects yeah so you have to move from static objects once you recognize them and have a very high probability of hitting the right targets with your neural net then you can move on to dynamic which makes it a lot more difficult because it's moving the data can look different and so answer brings an another level of complexity once you have solved that then you can go to the maximum level you need for autonomous driving yeah which is the real intelligence that you can start making predictions around around your environment so if you see a human and recognize it as a human can label it as a human

what is that person about to do yeah is this person moving at a quick speed slow speed is the person sitting somewhere is he standing and the same with cars and your other surroundings so that's like the maximum level we can go to and we are still in the in the middle of this this system so we don't want to keep everything to ourselves we are working on this but this is also yeah like an open shell to everybody who is interested in this kind of topic we are sending these sensors without all of these steps because we want to have spread it out and we want to democratize 3d sensing so everybody who is here or or interested in building up neural nets based on our data sets and then yeah please happy to talk afterwards that was quick yeah so here you can just see how many companies are interested in this technology and the capabilities what you can do with it so it's pretty it's pretty broad I mean we have many

customers on our list already many of them interested so there's a lot of things for us to do to service all these inquiries we are partnering with Nvidia for example for autonomous driving and neural nets working on that yeah I talked about in Rotten Infineon already and then some other partners the founders we are three founders my co-founders are the guys working on the technology which invented the sensing system myself and the business guy in the round and yeah represent the technology and then talk to customers and last but not least what is a road map for this year as ho yeah we're standing here today at rise of AI in about in in about two months in in July we are launching our first commercial product so now the test kit anymore but it like an openly available product which is mainly targeted at robotics mainly at robotics companies but can also be used in different in different areas then next level what we're currently working on is an autonomous driving development kit so

with several sensors which I'll a grenade the data that you can come up with something similar you have seen in the video and we're currently on a roadmap to start working on the commercialized automotive sensor and you talk about a sample as like the first step and that will be done by end of the year yeah on top of that we're currently starting to raise our series around the company has been financed by some Silicon Valley investors and German investors up to now with a total funding amount of around two million dollars yeah so if there any of it was in the room happy to follow up afterwards again oh yeah no I'm happy to take your questions Thanks all right then we have a first question Thank You Tuvia so the question is what does it actually have to do with machine learning I mean okay the applications you can build on top of it obviously but inside it doesn't look for me like any machine learning algorithm needed you can do this was

classic signal processing so as I said we're not in the we don't we haven't trained any big neural nets right now yeah so we're just starting off with that I think what is what is more what we want to bring over that we are building the cheapest 3d sensing technology where you can train which you can implement in almost every topic in smart building into consumer electronics into robots into cars where you can start training your nets on we are just starting with this and I think I yeah the the main point is that we are that we are delivering the basis now that we have reliable data but we have to do all these steps yeah with a typically are with a very new technology and I think just as you heard from Koenig's you need very good data yeah so about I think six to nine months ago we started really intensively working on this and came to the conclusion oh wow the data we thought was easy to handle with neural nets

is not that easy to handle so you have to take many prerequisites so it's 8090 percent before you can really then train your net so rather see it is an enabling technology for artificial intelligence at this point and we're moving on to the next level so we're not saying we are vai company and right now thanks a lot any other question from the audience cool there's one thank you very interesting product the question is do you see your product where do you see your product in the line of autonomous driving because soon a sensor you said yourself is good for five meters but if I'm driving I'm 5 meter I travel 5 meters in a half a second so if the sensor classifies a person the persons that if we talk about driving our data is being added on to the full sensory stack combined with lidar sensors camera sensors and so on that's why I said new field sensing and you currently don't have a proper technology which is able to cover that close

distance so at the close distance becomes important if you drive slower yeah so but every autonomous car has to start somewhere it has to stop somewhere and they're maneuvering situations and in all these situations you need full 360-degree coverage also of the clothes clothes to stand surroundings so it's a complementary technology to also the other technologies and not the single one yes please behind you is the mic Hey so famously Tesla is not using the lidar technology right they're using cameras for the self-driving but they are also using ultrasonic sensors or something they call ultrasonic sensors so is this something similar to what you are doing or oh yeah very interesting question so Tesla uses 12 1u ultrasound sensors I have to let's say due to NEA reasons I cannot talk about the details but let's say a company like Tesla is extremely interested in the level of data we can provide because that's what they need to properly have their autonomous driving features they are fully rolled out and this is one of the

missing important pieces right now yeah thanks for the talk and are you planning to expend like the frequency bands of that device to do like like real sound detection also passive sound detection sure so what does our system is pretty let's say Hardware agnostic it's really very software focused so we can use any kind of transducers so any kind of speaker and any kind of microphone and any type of frequency sure you have to adjust the hard way a little but in the back it's mainly adjusting the software so for sure you can also use it yeah maybe for hearing the sirens of a yeah of an ambulance in the city or like kids playing around so for these kinds of safety features and a lot of other topics I think are possible and and also from the images I saw they didn't look for humans they didn't look like they have much information rather than that there is something is that enough to do like some sort of image recognition image classification object classification

with that data that's exactly the part where it becomes tricky which is a tricky part for us what we found out that it's not we cannot just use typical approaches of optical data put it onto our data and it works but we have to like train new types of neural nets on that and have to see what are the specific data points or the specific patterns we can see in inside of that data and the longer you start looking at an object the more data we get the more shapes we get the more points we get the denser of the point cloud becomes but that is something you have to be to be executed on and see where we're going

Rise of AI 2027 · Berlin · May 2027

Be in the room for the next one.

Ten years of talks started as conversations in one curated room in Berlin. The next edition brings 300 selected leaders together in May 2027.