By Scott Simmie

The safety of pedestrians and cyclists is important in any major city.

Bike lanes and crosswalks are the most obvious signs that we create infrastructure for this purpose. But there are limits to what that infrastructure can do.

Winter, for example, creates additional threats. Ice and snow are the most obvious problems, but potholes and deep puddles can also cause havoc – particularly for cyclists. Snow can accumulate quickly in a storm – and it’s not uncommon for snow plowed from the roadway to sometimes spill over into bike lanes (and even on sidewalks).

That’s part of the reason why Indro Robotics has built an innovative solution that will be put to the test this winter. It’s called the Street Smart Robot, or SSR – and it’s been designed from the ground up to help ensure safe winter cycling.

Below: A Wikimedia Commons image, taken in Whitehorse by Anthony DeLorenzo. Cyclists in Canada are a hardy bunch.

STREET SMART ROBOT

The catalyst for the SSR is a research and development partnership fund called the Wintertech Development Program. According to the program website, Wintertech “supports Ontario small and medium enterprises (SMEs) and their partners to validate, test, prototype, and demonstrate new products and technologies designed to meet the unique demands of winter weather conditions.”

Wintertech is run by OVIN, the Ontario Vehicle Innovation Network. That’s a province of Ontario initiative which “capitalizes on the economic potential of advanced automotive technologies and smart mobility solutions such as connected and autonomous vehicles (CAVs), and electric and low-carbon vehicle technologies, while enabling the province’s transportation and infrastructure networks to plan for and adapt to this evolution.”

And while bicycles aren’t connected and autonomous vehicles (at least yet), we felt robotics could play a role in helping to ensure the safety of cyclists in the Smart City of the future. Specifically, we thought an autonomous robot equipped with the right sensors and processing might help to detect safety issues in bike lanes in the winter.

“The idea behind the robot is we want to prolong the use of bike lanes in Ottawa, but also ensure the safety of bike lanes in Ottawa,” explains Indro Robotics Account Executive Luke Corbeth.

“There’s really two parts to this: The first is a machine vision element to see if conditions are good enough for biking – no ice, not too many leaves, etc. On the safety side, the Street Smart Robot is more concerned with detecting things like potholes and cracks. And the idea is if you’re able to identify those things, the right resources can be deployed faster and more efficiently to solve the problem in a timely manner.”

So that’s the idea behind the SSR. Now, let’s take a look at the technology.

Below: A look at the specs of the AgileX Bunker Pro platform. We’ve upgraded the battery for a four-hour run time:

BUILT TOUGH

If you’re going to be out and about in an Ottawa winter – where temperatures have reached as low as -33°C – you need a platform suitable for the environment. InDro selected the AgileX Bunker Pro as the starting point.

With an IP67 Ingress Protection rating, the Bunker Pro is highly impervious to water and particulate matter – which it will encounter in abundance on Ottawa streets. The treads give an edge over wheels when it comes to navigating over ice, snow and potholes. With a full charge, the platform can run for some four hours – even when it’s cold out.

But the platform is just the start. InDro engineers had to put considerable thought into the kinds of sensors that would help the Street Smart Robot carry out its task while safely avoiding cyclists, pedestrians, or other obstacles it might encounter. And that began with four wide-angle pinhole cameras.

“The pinholes are just to give the operator a better understanding of their environment and situational awareness,” explains Corbeth. And while we say “operator” here, it might be more appropriate to say “observer.” Though initial deployments will involve teleoperation, the SSR has been built to carry out its tasks autonomously. We anticipate that down the road (or bike path), a person will simply be involved in monitoring missions in case human intervention is required.

That autonomy will be carried out by a combination of computing power, our InDro Autonomy software stack, and a plethora of sensors.

SENSORS, SENSORS, SENSORS

The Street Smart Robot is equipped with front- and rear-facing depth cameras. These cameras have two visual sensors each, placed roughly the same distance apart as human eyes. That separation – as in human vision – allows the SSR to perceive the world in depth. Onboard computation (we’ll get to that) calculates how far obstacles might be in the robot’s immediate surroundings, and whether it needs to slow down, stop or change course to avoid obstacles.

The depth cameras are supplemented by two 270° 2D LiDAR sensors – which are devoted almost exclusively to obstacle avoidance and safety. There are two additional 3D LiDARS, placed to assist with Simultaneous Localization and Mapping, as well as a GPS and IMU (Inertial Measurement Unit). There’s even a Range Finder that can detect how far an object is off the ground.

“If you have an overhanging tree branch, for example, you could see where it is and see if it obscures the bike lane,” says Corbeth. “We’ve never used a Range Finder before, but could see it would be useful in this application.”

In addition, there’s a thermal/30X optical Pan-Tilt-Zoom camera, complete with a windshield wiper for those sloppy days.

Below: Check out what the SSR has on board.

SENSING, AND MAKING SENSE OF, ITS ENVIRONMENT

All of those sensors are great. But they don’t do a lot on their own. They require software and computational power to give their data meaning. Here, the SSR has a Jetson ORIN AGX for EDGE computing. The ROS2 operating system software lives onboard in InDro Commander – a module we’ve created for rapid integration of sensors and 5G remote operations. Commander can be thought of as the glue that holds everything together.

But still, there’s the issue of making sense of the data. How will the Street Smart Robot identify ice, water, leaves, snow, potholes, debris etc? And how will it determine whether they are significant enough to require a City of Ottawa maintenance crew to take action?

Here, the SSR will be going to school. And by that, we mean a combination of Machine Vision and Machine Learning will teach it what’s worth reporting and what isn’t. One way is to take actual images you expect to see on the bike path and identify which ones are significant and which can be ignored.

“For example, you could mount a camera on an actual bicycle and then use that dataset to train the robot,” says Corbeth. “But there are a couple of approaches. There’s also obviously the simulation element. You can use specific engines that have high-fidelity graphics to simulate possible different environments in high resolution. And you can use that simulated data for training.”

THEN WHAT?

In the early stages, there will be a human operator at the helm. They’ll likely see some of those obstacles with the pinhole cameras, but from the start we anticipate that the onboard AI will trigger an alert for the operator. Once the operator confirms that the image is worthy of the attention of a maintenance crew, they will contact the city directly.

That’s how things will start. Once we’ve established a high confidence level that the Machine Vision is correctly identifying anomalies, that process can be automated. The SSR will be programmed to automatically send an alert to the city, complete with a captured image of the potential problem and GPS coordinates.

“It’s the idea of anomaly detection – detecting and flagging anomalies. The SSR will also likely be able to identify the severity of the anomaly on a one-to-five scale. And because you have the GPS onboard, the exact location of the problem can be relayed to the city.”

Below: The SSR, during its public unveiling at the TCXpo exhibition in June. Front and rear view, to show off all its bits…

ON THE SHOULDERS OF SENTINEL

Every technical achievement owes something to its predecessors. And that’s no different with the Street Smart Robot.

InDro Robotics previously built Sentinel, a workhorse for remote asset inspection. That robot incorporated an AgileX Bunker platform, an InDro Commander integration module for sensors and remote teleoperations, and several other sensors. It’s a great robot that has proven itself in the most demanding conditions.

The Street Smart Robot builds on the success and learnings of Sentinel with additional compute power, many more sensors, a more robust design and extended operational range. And that opens the SSR up to many other tasks beyond bicycle paths.

“That’s what really excites me about this project – the idea that this technology is transferable across other inspection verticals,” says Corbeth. “This particular robot could be used in agriculture, and the track platform is really good in mining. I could envision this exact same robot in those two verticals, and others.”

THE 5G CONNECTION

When dealing with multiple sensors in real-time, a solid data bandwidth is crucial. Here, SSR is outfitted with a high-speed modem and special antennaes for optimizing 5G and 4G signals. And while the Street Smart Robot can be operated over 4G, it’s 5G that offers the pipeline necessary for the data to really flow. InDro’s continuing partnership with Rogers will ensure that the SSR has access to Canada’s largest and most reliable 5G network.

Both InDro and Rogers are contributing a significant amount of development money as part of the OVIN Wintertech program. The total value of the project is estimated at $1,395,000. InDro Robotics and Rogers Communications are contributing two–thirds of the total project value ($780,000 and $150,000). Support from the Government of Ontario, through OVIN, totals $465,000 or one-third of the project. 

Below: Innovations in the InDro Sentinel paved the way for the Street Smart Robot. This image was taken at the opening of the Drone and Advanced Robot Training and Testing facility (DARTT) in June of 2023. That’s Luke Corbeth holding the microphone.

INDRO’S TAKE

We’re pretty excited about the SSR. As Luke Corbeth says: “It’s a bad-ass robot.”

And it is. But we’re also excited about the project itself.

The Smart City of the future will have connected, embedded sensors (including on robots) helping with everything from traffic flow to updated weather and road conditions. Cyclists are a big part of any major urban centre, and the prospect of helping to ensure their safety appeals to us. It’s also an opportunity to really hone in on Machine Vision and Machine Learning as InDro trains the SSR for its task.

“Smart Cities are on their way, and it’s great to be involved in yet another project as we build the capacity for the future,” says InDro Robotics CEO Philip Reece.

“The Street Smart Robot has an important role to play, and could well be the template for anomaly detection and road maintenance at scale in the future. We’re grateful to OVIN and its Wintertech program for selecting this project for development; we’re excited to see what the SSR can do.”

InDro plans to start testing the SSR during the winter, with the project stretching to June of 2024. We’ll be writing more once the robot hits the bike paths.