Re: [Ros-kinect] Generalized downsampling of depth image (was: Re: Is it possible to change the resolution of depth image?)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image (was: Re: Is it possible to change the resolution of depth image?)

Radu B. Rusu
Administrator
Patrick,

On 12/08/2010 11:52 PM, Patrick Bouffard wrote:
> I also found that the VoxelGrid filter used too much CPU for what I
> wanted to do, which was to just cut down on the sheer amount of point
> cloud data so there was some hope of a filter chain operating at ~10
> Hz on the quadrotor.

We weren't aware of this issue as we haven't had a chance to try this on a lower powered system. There might be room for
optimization on the VoxelGrid itself, but as I mentioned in my previous e-mail, we can try something like a PixelFilter
which takes into account he fact that we're working on an organized dataset, to do the downsampling.

> Anyway, I was going to go ahead and implement something with the
> existing ROS kinect driver before the OpenNI announcement but I'm not
> sure how that might change things. For example their documentation
> mentions a cropping feature that can cut down on CPU usage. I thought
> this might actually do some thing on the Kinect side of things, which
> would cut down not only on the CPU usage of whatever filtering
> pipeline is used but also the USB traffic, which on some (esp. small,
> light flying) computers may not be a trivial load on the CPU in its
> own right. I asked
> (http://groups.google.com/group/openkinect/browse_thread/thread/c8427e283ba8efc4)
> on the OpenKinect list about this but haven't heard any response.

Personally, I'll give up on the kinect driver and concentrate on the OpenNI driver. It's so much better in terms of many
things -- and I really want to start developing PCL stuff on top of these point clouds and not spend months in
developing lower level driver code. Primesense did a fantastic job with the OpenNI interfaces that I feel we should all
just use them and develop the next generation of perception systems on top, rather than trying to tweak the lower level
routines.

> Any thoughts on this idea? Now that there is this new driver, where
> would such a functionality now best be implemented (if indeed it would
> make sense at all based on whatever new functionalities this
> PrimeSense driver might be enabling).

We can definitely test the onboard downsampling functionality, but that might not give us what we want. In any case,
developing a PCL (nodelet) for a more efficient downsampling (taking into account the organized data structure) for
lower powered systems, makes a lot of sense to me, as it will be useful for a lot of other systems.


Cheers,
Radu.
--
http://pointclouds.org
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image (was: Re: Is it possible to change the resolution of depth image?)

Patrick Bouffard
Hi Radu,

On Thu, Dec 9, 2010 at 1:29 AM, Radu Bogdan Rusu <[hidden email]> wrote:

> Patrick,
>
> On 12/08/2010 11:52 PM, Patrick Bouffard wrote:
>>
>> I also found that the VoxelGrid filter used too much CPU for what I
>> wanted to do, which was to just cut down on the sheer amount of point
>> cloud data so there was some hope of a filter chain operating at ~10
>> Hz on the quadrotor.
>
> We weren't aware of this issue as we haven't had a chance to try this on a
> lower powered system. There might be room for optimization on the VoxelGrid
> itself, but as I mentioned in my previous e-mail, we can try something like
> a PixelFilter which takes into account he fact that we're working on an
> organized dataset, to do the downsampling.

That sounds good, but, if you want to use, say, only 10% of the 300K
points on every given iteration, then won't the loop inside
OpenNIDriver::publish ()  still be running 10x as many times as
necessary (plus 10x the memory requirements, 10x as much data to log
with rosbag if you want to do that, etc.).

It certainly sounds like your PixelFilter could achieve the
functionality I'd want to have. Still, I have to think that removing
the unwanted points as early as possible in the chain would produce
the best performance.

It might help if I had a correct understanding of the 'chain' in the
first place.. Here's what I'm surmising happens in the 'old' (!)
freenect/ROS-kinect chain, where the simple windowing scheme I'm using
happens in step 4:

1. Sensor acquires IR image and determines disparity. Buffer of size
N*sizeof(disparity measurement).
2. Sensor sends this buffer over USB.
3. Freenect driver receives this buffer, sends pointer to it to ROS
kinect driver.
4. ROS kinect driver does M operations on the buffer to produce the
PointCloud2 message, which is of size M*sizeof(PointXYZ), where M is
the number of points in the window (in what I've been doing M is
approx N/16)
5. PCL nodelet filters do cool stuff, all with M (or fewer) points

So I don't know quite how steps 1-4 work with the new driver, but if I
understand correctly in what you're suggesting, only at step 5 would
the number of points in play drop down to M.

>> Anyway, I was going to go ahead and implement something with the
>> existing ROS kinect driver before the OpenNI announcement but I'm not
>> sure how that might change things. For example their documentation
>> mentions a cropping feature that can cut down on CPU usage. I thought
>> this might actually do some thing on the Kinect side of things, which
>> would cut down not only on the CPU usage of whatever filtering
>> pipeline is used but also the USB traffic, which on some (esp. small,
>> light flying) computers may not be a trivial load on the CPU in its
>> own right. I asked
>>
>> (http://groups.google.com/group/openkinect/browse_thread/thread/c8427e283ba8efc4)
>> on the OpenKinect list about this but haven't heard any response.
>
> Personally, I'll give up on the kinect driver and concentrate on the OpenNI
> driver. It's so much better in terms of many things -- and I really want to
> start developing PCL stuff on top of these point clouds and not spend months
> in developing lower level driver code. Primesense did a fantastic job with
> the OpenNI interfaces that I feel we should all just use them and develop
> the next generation of perception systems on top, rather than trying to
> tweak the lower level routines.

I'm never a fan of tweaking things at the low level--unless that's
really the only way to improve performance..

>> Any thoughts on this idea? Now that there is this new driver, where
>> would such a functionality now best be implemented (if indeed it would
>> make sense at all based on whatever new functionalities this
>> PrimeSense driver might be enabling).
>
> We can definitely test the onboard downsampling functionality, but that
> might not give us what we want. In any case, developing a PCL (nodelet) for
> a more efficient downsampling (taking into account the organized data
> structure) for lower powered systems, makes a lot of sense to me, as it will
> be useful for a lot of other systems.

Yes, regardless of whatever turns out to be most efficient for the use
cases I have in mind, what you're describing should definitely be
widely useful.

Cheers,
Pat
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image

Radu B. Rusu
Administrator

On 12/09/2010 02:16 AM, Patrick Bouffard wrote:

>> We weren't aware of this issue as we haven't had a chance to try this on a
>> lower powered system. There might be room for optimization on the VoxelGrid
>> itself, but as I mentioned in my previous e-mail, we can try something like
>> a PixelFilter which takes into account he fact that we're working on an
>> organized dataset, to do the downsampling.
>
> That sounds good, but, if you want to use, say, only 10% of the 300K
> points on every given iteration, then won't the loop inside
> OpenNIDriver::publish ()  still be running 10x as many times as
> necessary (plus 10x the memory requirements, 10x as much data to log
> with rosbag if you want to do that, etc.).

Ah - ok I see your point. I agree. So we got two options:

  * we can see whether the hardware/OpenNI software downsampling works, and add that to the driver - easy
  * we can hijack the driver and provide a "stupid downsampling mode" in the driver. Basically skipping pixels. I don't
like it too much, and if we do it for 1 camera, we should do it for all cameras and have that part of some REP, where
this interface is consistent.

But I agree with you - if after profiling, we discover that things are slow, we should do something about it. Desperate
times call for desperate measures :)

> It certainly sounds like your PixelFilter could achieve the
> functionality I'd want to have. Still, I have to think that removing
> the unwanted points as early as possible in the chain would produce
> the best performance.

Right. Makes sense to do it at the driver level. Though again, we should profile this.

> It might help if I had a correct understanding of the 'chain' in the
> first place.. Here's what I'm surmising happens in the 'old' (!)
> freenect/ROS-kinect chain, where the simple windowing scheme I'm using
> happens in step 4:
>
> 1. Sensor acquires IR image and determines disparity. Buffer of size
> N*sizeof(disparity measurement).
> 2. Sensor sends this buffer over USB.
> 3. Freenect driver receives this buffer, sends pointer to it to ROS
> kinect driver.

Steps 1-3 replaced with a magic call on dept_.getDepthMap ().

> 4. ROS kinect driver does M operations on the buffer to produce the
> PointCloud2 message, which is of size M*sizeof(PointXYZ), where M is
> the number of points in the window (in what I've been doing M is
> approx N/16)

I see. So this has been greatly simplified as well. We're getting 3D from depth as:
         // Fill in XYZ
         pt_data[0] = (u - 320) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
         pt_data[1] = (v - 240) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
         pt_data[2] = depth_md_[k] * 0.001;

Further optimizations will follow (get some of that pixel_size_ * 0.001 / F_ precomputed outside of the loop). We do
this for (640 * 480 - invalid data). I don't expect this to be computationally intensive. If it is, we're screwed, as
nothing that we do on top of it could possibly be cheaper :)

> 5. PCL nodelet filters do cool stuff, all with M (or fewer) points

Right. A VoxelGrid with cutdown on Z at 3-4 meters (the data is unusable after that anyway). Alternatively a new
PixelFilter which is cheaper.

The advantage of a VoxelGrid is: "guaranteed" sampling density independent of the acquisition viewpoint. Very important
for a lot of processing blocks afterwards.


> So I don't know quite how steps 1-4 work with the new driver, but if I
> understand correctly in what you're suggesting, only at step 5 would
> the number of points in play drop down to M.

Yes. In my opinion, if we can't do the OpenNI hardware/software downsampling, we should try to do it outside of the
driver, at step 5... unless profiling the code shows otherwise (e.g., too much memory used, too much CPU used, etc)


Cheers,
Radu.
--
http://pointclouds.org
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image

Patrick Bouffard
Hi Radu,

So sounds like we're basically on the same page. I think the main
difference will be what is considered "too much {memory,CPU}" etc. as
that will have everything to do with hardware and use case. As you
say, that is best determined through testing.

I think the way forward (for me at least) will be first to move over
to the PrimeSense driver and then to do said testing. If your
PixelFilter is available soon that will be helpful! Actually, could
you describe in a bit more detail what PixelFilter can do? Would it be
able to do the sort of "generalized downsampling" (I know, probably
not the best term for it, suggestions welcome!) that I described.

Cheers,
Pat

On Thu, Dec 9, 2010 at 11:37 AM, Kurt Konolige
<[hidden email]> wrote:

> My $.02:
>
> 1. Allow downsampling at the device to 320x240, if it works - saves on
> USB bandwidth and driver processing
> 2. Allow further downsampling in the driver, using either decimation
> or "smart" binning, by which I mean check that you're not at a depth
> boundary when averaging.
>
> Note that if you want RGB registration to the depth, it is helpful to
> map RGB to the downsampled depth image, rather than the other way
> around, which would produce a lot of holes.  Alternatively, the RGB
> could be downsampled too.
>
> We'll look at supporting these in the driver.
>
> Cheers --Kurt
>
> On Thu, Dec 9, 2010 at 11:02 AM, Radu Bogdan Rusu <[hidden email]> wrote:
>>
>> On 12/09/2010 02:16 AM, Patrick Bouffard wrote:
>>>> We weren't aware of this issue as we haven't had a chance to try this on a
>>>> lower powered system. There might be room for optimization on the VoxelGrid
>>>> itself, but as I mentioned in my previous e-mail, we can try something like
>>>> a PixelFilter which takes into account he fact that we're working on an
>>>> organized dataset, to do the downsampling.
>>>
>>> That sounds good, but, if you want to use, say, only 10% of the 300K
>>> points on every given iteration, then won't the loop inside
>>> OpenNIDriver::publish ()  still be running 10x as many times as
>>> necessary (plus 10x the memory requirements, 10x as much data to log
>>> with rosbag if you want to do that, etc.).
>>
>> Ah - ok I see your point. I agree. So we got two options:
>>
>>  * we can see whether the hardware/OpenNI software downsampling works, and add that to the driver - easy
>>  * we can hijack the driver and provide a "stupid downsampling mode" in the driver. Basically skipping pixels. I don't
>> like it too much, and if we do it for 1 camera, we should do it for all cameras and have that part of some REP, where
>> this interface is consistent.
>>
>> But I agree with you - if after profiling, we discover that things are slow, we should do something about it. Desperate
>> times call for desperate measures :)
>>
>>> It certainly sounds like your PixelFilter could achieve the
>>> functionality I'd want to have. Still, I have to think that removing
>>> the unwanted points as early as possible in the chain would produce
>>> the best performance.
>>
>> Right. Makes sense to do it at the driver level. Though again, we should profile this.
>>
>>> It might help if I had a correct understanding of the 'chain' in the
>>> first place.. Here's what I'm surmising happens in the 'old' (!)
>>> freenect/ROS-kinect chain, where the simple windowing scheme I'm using
>>> happens in step 4:
>>>
>>> 1. Sensor acquires IR image and determines disparity. Buffer of size
>>> N*sizeof(disparity measurement).
>>> 2. Sensor sends this buffer over USB.
>>> 3. Freenect driver receives this buffer, sends pointer to it to ROS
>>> kinect driver.
>>
>> Steps 1-3 replaced with a magic call on dept_.getDepthMap ().
>>
>>> 4. ROS kinect driver does M operations on the buffer to produce the
>>> PointCloud2 message, which is of size M*sizeof(PointXYZ), where M is
>>> the number of points in the window (in what I've been doing M is
>>> approx N/16)
>>
>> I see. So this has been greatly simplified as well. We're getting 3D from depth as:
>>         // Fill in XYZ
>>         pt_data[0] = (u - 320) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
>>         pt_data[1] = (v - 240) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
>>         pt_data[2] = depth_md_[k] * 0.001;
>>
>> Further optimizations will follow (get some of that pixel_size_ * 0.001 / F_ precomputed outside of the loop). We do
>> this for (640 * 480 - invalid data). I don't expect this to be computationally intensive. If it is, we're screwed, as
>> nothing that we do on top of it could possibly be cheaper :)
>>
>>> 5. PCL nodelet filters do cool stuff, all with M (or fewer) points
>>
>> Right. A VoxelGrid with cutdown on Z at 3-4 meters (the data is unusable after that anyway). Alternatively a new
>> PixelFilter which is cheaper.
>>
>> The advantage of a VoxelGrid is: "guaranteed" sampling density independent of the acquisition viewpoint. Very important
>> for a lot of processing blocks afterwards.
>>
>>
>>> So I don't know quite how steps 1-4 work with the new driver, but if I
>>> understand correctly in what you're suggesting, only at step 5 would
>>> the number of points in play drop down to M.
>>
>> Yes. In my opinion, if we can't do the OpenNI hardware/software downsampling, we should try to do it outside of the
>> driver, at step 5... unless profiling the code shows otherwise (e.g., too much memory used, too much CPU used, etc)
>>
>>
>> Cheers,
>> Radu.
>> --
>> http://pointclouds.org
>> _______________________________________________
>> Ros-kinect mailing list
>> [hidden email]
>> https://code.ros.org/mailman/listinfo/ros-kinect
>>
>
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image

garratt
I would also be interested in _optional_ downsampling at the driver
level (maybe a rosparam)  

There is a lot that can be done at 320x240, but I wouldn't want to have
the driver to commit either way.

Garratt




On Thu, 2010-12-09 at 12:57 -0800, Patrick Bouffard wrote:

> Hi Radu,
>
> So sounds like we're basically on the same page. I think the main
> difference will be what is considered "too much {memory,CPU}" etc. as
> that will have everything to do with hardware and use case. As you
> say, that is best determined through testing.
>
> I think the way forward (for me at least) will be first to move over
> to the PrimeSense driver and then to do said testing. If your
> PixelFilter is available soon that will be helpful! Actually, could
> you describe in a bit more detail what PixelFilter can do? Would it be
> able to do the sort of "generalized downsampling" (I know, probably
> not the best term for it, suggestions welcome!) that I described.
>
> Cheers,
> Pat
>
> On Thu, Dec 9, 2010 at 11:37 AM, Kurt Konolige
> <[hidden email]> wrote:
> > My $.02:
> >
> > 1. Allow downsampling at the device to 320x240, if it works - saves on
> > USB bandwidth and driver processing
> > 2. Allow further downsampling in the driver, using either decimation
> > or "smart" binning, by which I mean check that you're not at a depth
> > boundary when averaging.
> >
> > Note that if you want RGB registration to the depth, it is helpful to
> > map RGB to the downsampled depth image, rather than the other way
> > around, which would produce a lot of holes.  Alternatively, the RGB
> > could be downsampled too.
> >
> > We'll look at supporting these in the driver.
> >
> > Cheers --Kurt
> >
> > On Thu, Dec 9, 2010 at 11:02 AM, Radu Bogdan Rusu <[hidden email]> wrote:
> >>
> >> On 12/09/2010 02:16 AM, Patrick Bouffard wrote:
> >>>> We weren't aware of this issue as we haven't had a chance to try this on a
> >>>> lower powered system. There might be room for optimization on the VoxelGrid
> >>>> itself, but as I mentioned in my previous e-mail, we can try something like
> >>>> a PixelFilter which takes into account he fact that we're working on an
> >>>> organized dataset, to do the downsampling.
> >>>
> >>> That sounds good, but, if you want to use, say, only 10% of the 300K
> >>> points on every given iteration, then won't the loop inside
> >>> OpenNIDriver::publish ()  still be running 10x as many times as
> >>> necessary (plus 10x the memory requirements, 10x as much data to log
> >>> with rosbag if you want to do that, etc.).
> >>
> >> Ah - ok I see your point. I agree. So we got two options:
> >>
> >>  * we can see whether the hardware/OpenNI software downsampling works, and add that to the driver - easy
> >>  * we can hijack the driver and provide a "stupid downsampling mode" in the driver. Basically skipping pixels. I don't
> >> like it too much, and if we do it for 1 camera, we should do it for all cameras and have that part of some REP, where
> >> this interface is consistent.
> >>
> >> But I agree with you - if after profiling, we discover that things are slow, we should do something about it. Desperate
> >> times call for desperate measures :)
> >>
> >>> It certainly sounds like your PixelFilter could achieve the
> >>> functionality I'd want to have. Still, I have to think that removing
> >>> the unwanted points as early as possible in the chain would produce
> >>> the best performance.
> >>
> >> Right. Makes sense to do it at the driver level. Though again, we should profile this.
> >>
> >>> It might help if I had a correct understanding of the 'chain' in the
> >>> first place.. Here's what I'm surmising happens in the 'old' (!)
> >>> freenect/ROS-kinect chain, where the simple windowing scheme I'm using
> >>> happens in step 4:
> >>>
> >>> 1. Sensor acquires IR image and determines disparity. Buffer of size
> >>> N*sizeof(disparity measurement).
> >>> 2. Sensor sends this buffer over USB.
> >>> 3. Freenect driver receives this buffer, sends pointer to it to ROS
> >>> kinect driver.
> >>
> >> Steps 1-3 replaced with a magic call on dept_.getDepthMap ().
> >>
> >>> 4. ROS kinect driver does M operations on the buffer to produce the
> >>> PointCloud2 message, which is of size M*sizeof(PointXYZ), where M is
> >>> the number of points in the window (in what I've been doing M is
> >>> approx N/16)
> >>
> >> I see. So this has been greatly simplified as well. We're getting 3D from depth as:
> >>         // Fill in XYZ
> >>         pt_data[0] = (u - 320) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
> >>         pt_data[1] = (v - 240) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
> >>         pt_data[2] = depth_md_[k] * 0.001;
> >>
> >> Further optimizations will follow (get some of that pixel_size_ * 0.001 / F_ precomputed outside of the loop). We do
> >> this for (640 * 480 - invalid data). I don't expect this to be computationally intensive. If it is, we're screwed, as
> >> nothing that we do on top of it could possibly be cheaper :)
> >>
> >>> 5. PCL nodelet filters do cool stuff, all with M (or fewer) points
> >>
> >> Right. A VoxelGrid with cutdown on Z at 3-4 meters (the data is unusable after that anyway). Alternatively a new
> >> PixelFilter which is cheaper.
> >>
> >> The advantage of a VoxelGrid is: "guaranteed" sampling density independent of the acquisition viewpoint. Very important
> >> for a lot of processing blocks afterwards.
> >>
> >>
> >>> So I don't know quite how steps 1-4 work with the new driver, but if I
> >>> understand correctly in what you're suggesting, only at step 5 would
> >>> the number of points in play drop down to M.
> >>
> >> Yes. In my opinion, if we can't do the OpenNI hardware/software downsampling, we should try to do it outside of the
> >> driver, at step 5... unless profiling the code shows otherwise (e.g., too much memory used, too much CPU used, etc)
> >>
> >>
> >> Cheers,
> >> Radu.
> >> --
> >> http://pointclouds.org
> >> _______________________________________________
> >> Ros-kinect mailing list
> >> [hidden email]
> >> https://code.ros.org/mailman/listinfo/ros-kinect
> >>
> >
> _______________________________________________
> [hidden email] / http://pointclouds.org
> https://code.ros.org/mailman/listinfo/pcl-users


_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image

Ivan Dryanovski
I played around with downsampling using a voxel grid. I'm running on a
2Ghz Intel Core 2 Quad. For a 640x480 cloud published by the kinect,

I am using the following code to time a downsample:

sensor_msgs::PointCloud2 cloudMsgFiltered;
pcl::VoxelGrid<sensor_msgs::PointCloud2> sor;
sor.setInputCloud (cloudMsg);
sor.setLeafSize (size_, size_, size_);
sor.filter (cloudMsgFiltered);

 I'm getting the following downsample times, with the openni driver
running live:

~350 ms, leaf size = 0.01
~ 100 ms, leaf size = 0.10

Do these look reasonable?

Thanks,
Ivan

On Thu, Dec 9, 2010 at 4:50 PM, garratt <[hidden email]> wrote:

> I would also be interested in _optional_ downsampling at the driver
> level (maybe a rosparam)
>
> There is a lot that can be done at 320x240, but I wouldn't want to have
> the driver to commit either way.
>
> Garratt
>
>
>
>
> On Thu, 2010-12-09 at 12:57 -0800, Patrick Bouffard wrote:
>> Hi Radu,
>>
>> So sounds like we're basically on the same page. I think the main
>> difference will be what is considered "too much {memory,CPU}" etc. as
>> that will have everything to do with hardware and use case. As you
>> say, that is best determined through testing.
>>
>> I think the way forward (for me at least) will be first to move over
>> to the PrimeSense driver and then to do said testing. If your
>> PixelFilter is available soon that will be helpful! Actually, could
>> you describe in a bit more detail what PixelFilter can do? Would it be
>> able to do the sort of "generalized downsampling" (I know, probably
>> not the best term for it, suggestions welcome!) that I described.
>>
>> Cheers,
>> Pat
>>
>> On Thu, Dec 9, 2010 at 11:37 AM, Kurt Konolige
>> <[hidden email]> wrote:
>> > My $.02:
>> >
>> > 1. Allow downsampling at the device to 320x240, if it works - saves on
>> > USB bandwidth and driver processing
>> > 2. Allow further downsampling in the driver, using either decimation
>> > or "smart" binning, by which I mean check that you're not at a depth
>> > boundary when averaging.
>> >
>> > Note that if you want RGB registration to the depth, it is helpful to
>> > map RGB to the downsampled depth image, rather than the other way
>> > around, which would produce a lot of holes.  Alternatively, the RGB
>> > could be downsampled too.
>> >
>> > We'll look at supporting these in the driver.
>> >
>> > Cheers --Kurt
>> >
>> > On Thu, Dec 9, 2010 at 11:02 AM, Radu Bogdan Rusu <[hidden email]> wrote:
>> >>
>> >> On 12/09/2010 02:16 AM, Patrick Bouffard wrote:
>> >>>> We weren't aware of this issue as we haven't had a chance to try this on a
>> >>>> lower powered system. There might be room for optimization on the VoxelGrid
>> >>>> itself, but as I mentioned in my previous e-mail, we can try something like
>> >>>> a PixelFilter which takes into account he fact that we're working on an
>> >>>> organized dataset, to do the downsampling.
>> >>>
>> >>> That sounds good, but, if you want to use, say, only 10% of the 300K
>> >>> points on every given iteration, then won't the loop inside
>> >>> OpenNIDriver::publish ()  still be running 10x as many times as
>> >>> necessary (plus 10x the memory requirements, 10x as much data to log
>> >>> with rosbag if you want to do that, etc.).
>> >>
>> >> Ah - ok I see your point. I agree. So we got two options:
>> >>
>> >>  * we can see whether the hardware/OpenNI software downsampling works, and add that to the driver - easy
>> >>  * we can hijack the driver and provide a "stupid downsampling mode" in the driver. Basically skipping pixels. I don't
>> >> like it too much, and if we do it for 1 camera, we should do it for all cameras and have that part of some REP, where
>> >> this interface is consistent.
>> >>
>> >> But I agree with you - if after profiling, we discover that things are slow, we should do something about it. Desperate
>> >> times call for desperate measures :)
>> >>
>> >>> It certainly sounds like your PixelFilter could achieve the
>> >>> functionality I'd want to have. Still, I have to think that removing
>> >>> the unwanted points as early as possible in the chain would produce
>> >>> the best performance.
>> >>
>> >> Right. Makes sense to do it at the driver level. Though again, we should profile this.
>> >>
>> >>> It might help if I had a correct understanding of the 'chain' in the
>> >>> first place.. Here's what I'm surmising happens in the 'old' (!)
>> >>> freenect/ROS-kinect chain, where the simple windowing scheme I'm using
>> >>> happens in step 4:
>> >>>
>> >>> 1. Sensor acquires IR image and determines disparity. Buffer of size
>> >>> N*sizeof(disparity measurement).
>> >>> 2. Sensor sends this buffer over USB.
>> >>> 3. Freenect driver receives this buffer, sends pointer to it to ROS
>> >>> kinect driver.
>> >>
>> >> Steps 1-3 replaced with a magic call on dept_.getDepthMap ().
>> >>
>> >>> 4. ROS kinect driver does M operations on the buffer to produce the
>> >>> PointCloud2 message, which is of size M*sizeof(PointXYZ), where M is
>> >>> the number of points in the window (in what I've been doing M is
>> >>> approx N/16)
>> >>
>> >> I see. So this has been greatly simplified as well. We're getting 3D from depth as:
>> >>         // Fill in XYZ
>> >>         pt_data[0] = (u - 320) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
>> >>         pt_data[1] = (v - 240) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
>> >>         pt_data[2] = depth_md_[k] * 0.001;
>> >>
>> >> Further optimizations will follow (get some of that pixel_size_ * 0.001 / F_ precomputed outside of the loop). We do
>> >> this for (640 * 480 - invalid data). I don't expect this to be computationally intensive. If it is, we're screwed, as
>> >> nothing that we do on top of it could possibly be cheaper :)
>> >>
>> >>> 5. PCL nodelet filters do cool stuff, all with M (or fewer) points
>> >>
>> >> Right. A VoxelGrid with cutdown on Z at 3-4 meters (the data is unusable after that anyway). Alternatively a new
>> >> PixelFilter which is cheaper.
>> >>
>> >> The advantage of a VoxelGrid is: "guaranteed" sampling density independent of the acquisition viewpoint. Very important
>> >> for a lot of processing blocks afterwards.
>> >>
>> >>
>> >>> So I don't know quite how steps 1-4 work with the new driver, but if I
>> >>> understand correctly in what you're suggesting, only at step 5 would
>> >>> the number of points in play drop down to M.
>> >>
>> >> Yes. In my opinion, if we can't do the OpenNI hardware/software downsampling, we should try to do it outside of the
>> >> driver, at step 5... unless profiling the code shows otherwise (e.g., too much memory used, too much CPU used, etc)
>> >>
>> >>
>> >> Cheers,
>> >> Radu.
>> >> --
>> >> http://pointclouds.org
>> >> _______________________________________________
>> >> Ros-kinect mailing list
>> >> [hidden email]
>> >> https://code.ros.org/mailman/listinfo/ros-kinect
>> >>
>> >
>> _______________________________________________
>> [hidden email] / http://pointclouds.org
>> https://code.ros.org/mailman/listinfo/pcl-users
>
>
> _______________________________________________
> [hidden email] / http://pointclouds.org
> https://code.ros.org/mailman/listinfo/pcl-users
>
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image

Radu B. Rusu
Administrator


On 12/10/2010 04:53 PM, Ivan Dryanovski wrote:

> I played around with downsampling using a voxel grid. I'm running on a
> 2Ghz Intel Core 2 Quad. For a 640x480 cloud published by the kinect,
>
> I am using the following code to time a downsample:
>
> sensor_msgs::PointCloud2 cloudMsgFiltered;
> pcl::VoxelGrid<sensor_msgs::PointCloud2>  sor;
> sor.setInputCloud (cloudMsg);
> sor.setLeafSize (size_, size_, size_);
> sor.filter (cloudMsgFiltered);
>
>   I'm getting the following downsample times, with the openni driver
> running live:
>
> ~350 ms, leaf size = 0.01
> ~ 100 ms, leaf size = 0.10
>
> Do these look reasonable?
>
> Thanks,
> Ivan

Ivan,

Can you make a minimal example with a NodeHandle that subscribes to a PointCloud2 topic, applies that, times it, and
publishes it afterwards ?

That way I can test it really quickly on some machines here to provide some timings, and then spend some time seeing
whether we can optimize it even further.

Thanks!

Cheers,
Radu.
--
http://pointclouds.org
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: [Ros-kinect] Generalized downsampling of depth image

Radu B. Rusu
Administrator
In reply to this post by Ivan Dryanovski
Thanks Ivan, got the file.

Ok, I get ~100ms on 0,01 and 50-60ms on 0.1 on my laptop, with these two additions:
filter.setFilterFieldName("z");
filter.setFilterLimits(0.01, 3.0);

My motivation for that is that at distances higher than 3m, the depth data is unusable anyway, but your mileage might vary.

Same tests on a proper i7 desktop:

~39-40ms for 0.1
~60-62ms for 0.01

CPU usage seems to be consistent at ~15% for 1 core (we're doing deserialization + copy too).

Let's see if we can optimize these even further. Again, VoxelGrid is supposed to be the generic way of doing this...
I'll try to finish and commit the PixelFilter soon for comparison.

In any case, a dumb +=2 in the driver will reduce the number of points way faster than anything else :) The problem is
when you need the camera image downsampled too - then you start running into problems... unless you use an algorithm
such as the ones in OpenCV you might end up with a crappy RGB image. But for now, we can just add a dynamic_reconfigure
option... funny how we were complaining when we had crappy 176x144 resolutions from other TOF cameras, and now that we
got VGA we wanna go back :)

Cheers,
Radu.
--
http://pointclouds.org

On 12/10/2010 04:53 PM, Ivan Dryanovski wrote:

> I played around with downsampling using a voxel grid. I'm running on a
> 2Ghz Intel Core 2 Quad. For a 640x480 cloud published by the kinect,
>
> I am using the following code to time a downsample:
>
> sensor_msgs::PointCloud2 cloudMsgFiltered;
> pcl::VoxelGrid<sensor_msgs::PointCloud2>  sor;
> sor.setInputCloud (cloudMsg);
> sor.setLeafSize (size_, size_, size_);
> sor.filter (cloudMsgFiltered);
>
>   I'm getting the following downsample times, with the openni driver
> running live:
>
> ~350 ms, leaf size = 0.01
> ~ 100 ms, leaf size = 0.10
>
> Do these look reasonable?
>
> Thanks,
> Ivan
>
> On Thu, Dec 9, 2010 at 4:50 PM, garratt<[hidden email]>  wrote:
>> I would also be interested in _optional_ downsampling at the driver
>> level (maybe a rosparam)
>>
>> There is a lot that can be done at 320x240, but I wouldn't want to have
>> the driver to commit either way.
>>
>> Garratt
>>
>>
>>
>>
>> On Thu, 2010-12-09 at 12:57 -0800, Patrick Bouffard wrote:
>>> Hi Radu,
>>>
>>> So sounds like we're basically on the same page. I think the main
>>> difference will be what is considered "too much {memory,CPU}" etc. as
>>> that will have everything to do with hardware and use case. As you
>>> say, that is best determined through testing.
>>>
>>> I think the way forward (for me at least) will be first to move over
>>> to the PrimeSense driver and then to do said testing. If your
>>> PixelFilter is available soon that will be helpful! Actually, could
>>> you describe in a bit more detail what PixelFilter can do? Would it be
>>> able to do the sort of "generalized downsampling" (I know, probably
>>> not the best term for it, suggestions welcome!) that I described.
>>>
>>> Cheers,
>>> Pat
>>>
>>> On Thu, Dec 9, 2010 at 11:37 AM, Kurt Konolige
>>> <[hidden email]>  wrote:
>>>> My $.02:
>>>>
>>>> 1. Allow downsampling at the device to 320x240, if it works - saves on
>>>> USB bandwidth and driver processing
>>>> 2. Allow further downsampling in the driver, using either decimation
>>>> or "smart" binning, by which I mean check that you're not at a depth
>>>> boundary when averaging.
>>>>
>>>> Note that if you want RGB registration to the depth, it is helpful to
>>>> map RGB to the downsampled depth image, rather than the other way
>>>> around, which would produce a lot of holes.  Alternatively, the RGB
>>>> could be downsampled too.
>>>>
>>>> We'll look at supporting these in the driver.
>>>>
>>>> Cheers --Kurt
>>>>
>>>> On Thu, Dec 9, 2010 at 11:02 AM, Radu Bogdan Rusu<[hidden email]>  wrote:
>>>>>
>>>>> On 12/09/2010 02:16 AM, Patrick Bouffard wrote:
>>>>>>> We weren't aware of this issue as we haven't had a chance to try this on a
>>>>>>> lower powered system. There might be room for optimization on the VoxelGrid
>>>>>>> itself, but as I mentioned in my previous e-mail, we can try something like
>>>>>>> a PixelFilter which takes into account he fact that we're working on an
>>>>>>> organized dataset, to do the downsampling.
>>>>>>
>>>>>> That sounds good, but, if you want to use, say, only 10% of the 300K
>>>>>> points on every given iteration, then won't the loop inside
>>>>>> OpenNIDriver::publish ()  still be running 10x as many times as
>>>>>> necessary (plus 10x the memory requirements, 10x as much data to log
>>>>>> with rosbag if you want to do that, etc.).
>>>>>
>>>>> Ah - ok I see your point. I agree. So we got two options:
>>>>>
>>>>>   * we can see whether the hardware/OpenNI software downsampling works, and add that to the driver - easy
>>>>>   * we can hijack the driver and provide a "stupid downsampling mode" in the driver. Basically skipping pixels. I don't
>>>>> like it too much, and if we do it for 1 camera, we should do it for all cameras and have that part of some REP, where
>>>>> this interface is consistent.
>>>>>
>>>>> But I agree with you - if after profiling, we discover that things are slow, we should do something about it. Desperate
>>>>> times call for desperate measures :)
>>>>>
>>>>>> It certainly sounds like your PixelFilter could achieve the
>>>>>> functionality I'd want to have. Still, I have to think that removing
>>>>>> the unwanted points as early as possible in the chain would produce
>>>>>> the best performance.
>>>>>
>>>>> Right. Makes sense to do it at the driver level. Though again, we should profile this.
>>>>>
>>>>>> It might help if I had a correct understanding of the 'chain' in the
>>>>>> first place.. Here's what I'm surmising happens in the 'old' (!)
>>>>>> freenect/ROS-kinect chain, where the simple windowing scheme I'm using
>>>>>> happens in step 4:
>>>>>>
>>>>>> 1. Sensor acquires IR image and determines disparity. Buffer of size
>>>>>> N*sizeof(disparity measurement).
>>>>>> 2. Sensor sends this buffer over USB.
>>>>>> 3. Freenect driver receives this buffer, sends pointer to it to ROS
>>>>>> kinect driver.
>>>>>
>>>>> Steps 1-3 replaced with a magic call on dept_.getDepthMap ().
>>>>>
>>>>>> 4. ROS kinect driver does M operations on the buffer to produce the
>>>>>> PointCloud2 message, which is of size M*sizeof(PointXYZ), where M is
>>>>>> the number of points in the window (in what I've been doing M is
>>>>>> approx N/16)
>>>>>
>>>>> I see. So this has been greatly simplified as well. We're getting 3D from depth as:
>>>>>          // Fill in XYZ
>>>>>          pt_data[0] = (u - 320) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
>>>>>          pt_data[1] = (v - 240) * pixel_size_ * depth_md_[k] * 0.001 / F_ ;
>>>>>          pt_data[2] = depth_md_[k] * 0.001;
>>>>>
>>>>> Further optimizations will follow (get some of that pixel_size_ * 0.001 / F_ precomputed outside of the loop). We do
>>>>> this for (640 * 480 - invalid data). I don't expect this to be computationally intensive. If it is, we're screwed, as
>>>>> nothing that we do on top of it could possibly be cheaper :)
>>>>>
>>>>>> 5. PCL nodelet filters do cool stuff, all with M (or fewer) points
>>>>>
>>>>> Right. A VoxelGrid with cutdown on Z at 3-4 meters (the data is unusable after that anyway). Alternatively a new
>>>>> PixelFilter which is cheaper.
>>>>>
>>>>> The advantage of a VoxelGrid is: "guaranteed" sampling density independent of the acquisition viewpoint. Very important
>>>>> for a lot of processing blocks afterwards.
>>>>>
>>>>>
>>>>>> So I don't know quite how steps 1-4 work with the new driver, but if I
>>>>>> understand correctly in what you're suggesting, only at step 5 would
>>>>>> the number of points in play drop down to M.
>>>>>
>>>>> Yes. In my opinion, if we can't do the OpenNI hardware/software downsampling, we should try to do it outside of the
>>>>> driver, at step 5... unless profiling the code shows otherwise (e.g., too much memory used, too much CPU used, etc)
>>>>>
>>>>>
>>>>> Cheers,
>>>>> Radu.
>>>>> --
>>>>> http://pointclouds.org
>>>>> _______________________________________________
>>>>> Ros-kinect mailing list
>>>>> [hidden email]
>>>>> https://code.ros.org/mailman/listinfo/ros-kinect
>>>>>
>>>>
>>> _______________________________________________
>>> [hidden email] / http://pointclouds.org
>>> https://code.ros.org/mailman/listinfo/pcl-users
>>
>>
>> _______________________________________________
>> [hidden email] / http://pointclouds.org
>> https://code.ros.org/mailman/listinfo/pcl-users
>>
> _______________________________________________
> [hidden email] / http://pointclouds.org
> https://code.ros.org/mailman/listinfo/pcl-users
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users