surfels and PCL

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

surfels and PCL

Stéphane Magnenat
Hello,

In the continuation of our work with the kinect, we are interested at
looking into combining multiple scans using surfels. Surfels [1] are
oriented surface elements (point+normal+radius), and typically hold
color information and view statistics, to provide intelligence to the
update process. Recent works [2,3] tend to indicate that surfels are
well-suited for RGB-D sensors such as the kinect.

We are interested in adding surfel support to ros/pcl/rviz. To that end,
it would be nice to agree on a naming convention for surfels (field
names in PointCloud2), to add a corresponding type to
pcl/point_types.hpp, and to add support in rviz. As far as I can see,
there is already support for PointXYZRGBNormal in PCL. The minimum to
add for surfel support would be a radius. The view statistics (8x8 bins,
8-bit histogram) might be interesting as well, but I am not sure yet
whether it is really useful to put the statistics in the point cloud itself.

What do you think? Is there interest for surfels in the PCL community?

Thank you, kind regards,

Stéphane


[1] Pfister et al., Surfels: surface elements as rendering primitives
http://www.cg.inf.ethz.ch/Downloads/Publications/Papers/2000/p_Pfi00.pdf

[2] T Weise et al., In-hand Scanning with Online Loop Closure
http://www.mmp.rwth-aachen.de/publications/pdf/weise-onlineloopclosure-3dim09.pdf

[3] Henry et al. - RGB-D Mapping: Using Depth Cameras for Dense 3D
Modeling of Indoor Environments
http://www.seattle.intel-research.net/RGBD/RGBD-RSS2010/papers/henry-RGBD10-RGBD-mapping.pdf 


--
Dr Stéphane Magnenat
http://stephane.magnenat.net
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Radu B. Rusu
Administrator
Stéphane,

Can we give it a try with PointXYZRGBNormal + radius and see where that takes us? We can name it Surfel or anything else
for that matter (PointSurfel?).


Cheers,
Radu.
--
http://pointclouds.org

On 02/02/2011 11:33 AM, Stéphane Magnenat wrote:

> Hello,
>
> In the continuation of our work with the kinect, we are interested at
> looking into combining multiple scans using surfels. Surfels [1] are
> oriented surface elements (point+normal+radius), and typically hold
> color information and view statistics, to provide intelligence to the
> update process. Recent works [2,3] tend to indicate that surfels are
> well-suited for RGB-D sensors such as the kinect.
>
> We are interested in adding surfel support to ros/pcl/rviz. To that end,
> it would be nice to agree on a naming convention for surfels (field
> names in PointCloud2), to add a corresponding type to
> pcl/point_types.hpp, and to add support in rviz. As far as I can see,
> there is already support for PointXYZRGBNormal in PCL. The minimum to
> add for surfel support would be a radius. The view statistics (8x8 bins,
> 8-bit histogram) might be interesting as well, but I am not sure yet
> whether it is really useful to put the statistics in the point cloud itself.
>
> What do you think? Is there interest for surfels in the PCL community?
>
> Thank you, kind regards,
>
> Stéphane
>
>
> [1] Pfister et al., Surfels: surface elements as rendering primitives
> http://www.cg.inf.ethz.ch/Downloads/Publications/Papers/2000/p_Pfi00.pdf
>
> [2] T Weise et al., In-hand Scanning with Online Loop Closure
> http://www.mmp.rwth-aachen.de/publications/pdf/weise-onlineloopclosure-3dim09.pdf
>
> [3] Henry et al. - RGB-D Mapping: Using Depth Cameras for Dense 3D
> Modeling of Indoor Environments
> http://www.seattle.intel-research.net/RGBD/RGBD-RSS2010/papers/henry-RGBD10-RGBD-mapping.pdf
>
>
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Nicolas Burrus
Hi,

I am also very interested by a surfel implementation in PCL. I have my
own quick'n'dirty implementation in rgbdemo that I would be happy to
get rid of :-)

However, to be useful surfels typically require a number of associated
data, such as the number of views in which the surfel appears, some
confidence values, maybe some weights to compute weighted running
average, etc. How could this be managed? Through pointers to
associated data? Would it be possible to get a
Surfel<AssociatedDataType> point type? Or this would be too complex
for further serialization?

Cheers,
Nicolas

On Wed, Feb 2, 2011 at 9:49 PM, Radu Bogdan Rusu <[hidden email]> wrote:

> Stéphane,
>
> Can we give it a try with PointXYZRGBNormal + radius and see where that takes us? We can name it Surfel or anything else
> for that matter (PointSurfel?).
>
>
> Cheers,
> Radu.
> --
> http://pointclouds.org
>
> On 02/02/2011 11:33 AM, Stéphane Magnenat wrote:
>> Hello,
>>
>> In the continuation of our work with the kinect, we are interested at
>> looking into combining multiple scans using surfels. Surfels [1] are
>> oriented surface elements (point+normal+radius), and typically hold
>> color information and view statistics, to provide intelligence to the
>> update process. Recent works [2,3] tend to indicate that surfels are
>> well-suited for RGB-D sensors such as the kinect.
>>
>> We are interested in adding surfel support to ros/pcl/rviz. To that end,
>> it would be nice to agree on a naming convention for surfels (field
>> names in PointCloud2), to add a corresponding type to
>> pcl/point_types.hpp, and to add support in rviz. As far as I can see,
>> there is already support for PointXYZRGBNormal in PCL. The minimum to
>> add for surfel support would be a radius. The view statistics (8x8 bins,
>> 8-bit histogram) might be interesting as well, but I am not sure yet
>> whether it is really useful to put the statistics in the point cloud itself.
>>
>> What do you think? Is there interest for surfels in the PCL community?
>>
>> Thank you, kind regards,
>>
>> Stéphane
>>
>>
>> [1] Pfister et al., Surfels: surface elements as rendering primitives
>> http://www.cg.inf.ethz.ch/Downloads/Publications/Papers/2000/p_Pfi00.pdf
>>
>> [2] T Weise et al., In-hand Scanning with Online Loop Closure
>> http://www.mmp.rwth-aachen.de/publications/pdf/weise-onlineloopclosure-3dim09.pdf
>>
>> [3] Henry et al. - RGB-D Mapping: Using Depth Cameras for Dense 3D
>> Modeling of Indoor Environments
>> http://www.seattle.intel-research.net/RGBD/RGBD-RSS2010/papers/henry-RGBD10-RGBD-mapping.pdf
>>
>>
> _______________________________________________
> [hidden email] / http://pointclouds.org
> https://code.ros.org/mailman/listinfo/pcl-users
>
>
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Michael Krainin
In reply to this post by Stéphane Magnenat
I just thought I'd chime in with my experience with ROS + surfels. We
(at UW + Intel Seattle) have been essentially using 3 separate data
structures (depending on whether we're doing surfel updates,
sending/saving data, or visualizing).

As Nicolas notes, there can be associated data with surfels that you
don't necessarily always want to pass around, so we have a Surfel
structure that has all of these. We use this structure when, for
instance, we're updating a surfel and want to know if we've already
seen it from that angle.

To pass surfels around when it's not crucial to have all of the
associated data, we use a point type defined as:
struct surfelPt
{
        PCL_ADD_POINT4D;    // This adds the members x,y,z which can also be
accessed using the point (which is float[4])
        float rgb; //packed
        PCL_ADD_NORMAL4D;   // This adds the member normal[3] which can also
be accessed using the point (which is float[4])
        float curvature;
        float radius;
        float confidence;
        EIGEN_MAKE_ALIGNED_OPERATOR_NEW
} EIGEN_ALIGN16;

To display surfels in rviz, we either use a PointCloud2 (if we don't
care about seeing them as discs) or we have a function to convert a
surfel cloud into a visualization_msgs::Marker. Each surfel in that
Marker is represented by 4 triangles.

-Mike

Author: Nicolas Burrus
Date: 2011-02-02 13:04 -800
To: Radu Bogdan Rusu, Point Cloud Library (PCL) mailing list
Subject: Re: [PCL-users] surfels and PCL
Hi,

I am also very interested by a surfel implementation in PCL. I have my
own quick'n'dirty implementation in rgbdemo that I would be happy to
get rid of :-)

However, to be useful surfels typically require a number of associated
data, such as the number of views in which the surfel appears, some
confidence values, maybe some weights to compute weighted running
average, etc. How could this be managed? Through pointers to
associated data? Would it be possible to get a
Surfel<AssociatedDataType> point type? Or this would be too complex
for further serialization?

Cheers,
Nicolas
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Peter Henry
I would add that we do also manage some data separately, such as the
"bins" that collect a histogram of directions the surfel has been
viewed from.  Though there's nothing fundamental that says we couldn't
store such (fixed size) data alongside the other members of the
surfelPt.

-Peter

On Wed, Feb 2, 2011 at 1:42 PM, Michael Krainin
<[hidden email]> wrote:

> I just thought I'd chime in with my experience with ROS + surfels. We
> (at UW + Intel Seattle) have been essentially using 3 separate data
> structures (depending on whether we're doing surfel updates,
> sending/saving data, or visualizing).
>
> As Nicolas notes, there can be associated data with surfels that you
> don't necessarily always want to pass around, so we have a Surfel
> structure that has all of these. We use this structure when, for
> instance, we're updating a surfel and want to know if we've already
> seen it from that angle.
>
> To pass surfels around when it's not crucial to have all of the
> associated data, we use a point type defined as:
> struct surfelPt
> {
>        PCL_ADD_POINT4D;    // This adds the members x,y,z which can also be
> accessed using the point (which is float[4])
>        float rgb; //packed
>        PCL_ADD_NORMAL4D;   // This adds the member normal[3] which can also
> be accessed using the point (which is float[4])
>        float curvature;
>        float radius;
>        float confidence;
>        EIGEN_MAKE_ALIGNED_OPERATOR_NEW
> } EIGEN_ALIGN16;
>
> To display surfels in rviz, we either use a PointCloud2 (if we don't
> care about seeing them as discs) or we have a function to convert a
> surfel cloud into a visualization_msgs::Marker. Each surfel in that
> Marker is represented by 4 triangles.
>
> -Mike
>
> Author: Nicolas Burrus
> Date: 2011-02-02 13:04 -800
> To: Radu Bogdan Rusu, Point Cloud Library (PCL) mailing list
> Subject: Re: [PCL-users] surfels and PCL
> Hi,
>
> I am also very interested by a surfel implementation in PCL. I have my
> own quick'n'dirty implementation in rgbdemo that I would be happy to
> get rid of :-)
>
> However, to be useful surfels typically require a number of associated
> data, such as the number of views in which the surfel appears, some
> confidence values, maybe some weights to compute weighted running
> average, etc. How could this be managed? Through pointers to
> associated data? Would it be possible to get a
> Surfel<AssociatedDataType> point type? Or this would be too complex
> for further serialization?
>
> Cheers,
> Nicolas
>



--
Peter Henry
[hidden email]
206-225-8275
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Peter Henry
In reply to this post by Michael Krainin
I would add that we do also manage some data separately, such as the
"bins" that collect a histogram of directions the surfel has been
viewed from.  Though there's nothing fundamental that says we couldn't
store such (fixed size) data alongside the other members of the
surfelPt.

-Peter

On Wed, Feb 2, 2011 at 1:42 PM, Michael Krainin
<[hidden email]> wrote:

> I just thought I'd chime in with my experience with ROS + surfels. We
> (at UW + Intel Seattle) have been essentially using 3 separate data
> structures (depending on whether we're doing surfel updates,
> sending/saving data, or visualizing).
>
> As Nicolas notes, there can be associated data with surfels that you
> don't necessarily always want to pass around, so we have a Surfel
> structure that has all of these. We use this structure when, for
> instance, we're updating a surfel and want to know if we've already
> seen it from that angle.
>
> To pass surfels around when it's not crucial to have all of the
> associated data, we use a point type defined as:
> struct surfelPt
> {
>        PCL_ADD_POINT4D;    // This adds the members x,y,z which can also be
> accessed using the point (which is float[4])
>        float rgb; //packed
>        PCL_ADD_NORMAL4D;   // This adds the member normal[3] which can also
> be accessed using the point (which is float[4])
>        float curvature;
>        float radius;
>        float confidence;
>        EIGEN_MAKE_ALIGNED_OPERATOR_NEW
> } EIGEN_ALIGN16;
>
> To display surfels in rviz, we either use a PointCloud2 (if we don't
> care about seeing them as discs) or we have a function to convert a
> surfel cloud into a visualization_msgs::Marker. Each surfel in that
> Marker is represented by 4 triangles.
>
> -Mike
>
> Author: Nicolas Burrus
> Date: 2011-02-02 13:04 -800
> To: Radu Bogdan Rusu, Point Cloud Library (PCL) mailing list
> Subject: Re: [PCL-users] surfels and PCL
> Hi,
>
> I am also very interested by a surfel implementation in PCL. I have my
> own quick'n'dirty implementation in rgbdemo that I would be happy to
> get rid of :-)
>
> However, to be useful surfels typically require a number of associated
> data, such as the number of views in which the surfel appears, some
> confidence values, maybe some weights to compute weighted running
> average, etc. How could this be managed? Through pointers to
> associated data? Would it be possible to get a
> Surfel<AssociatedDataType> point type? Or this would be too complex
> for further serialization?
>
> Cheers,
> Nicolas
>



--
Peter Henry
[hidden email]
206-225-8275
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Stéphane Magnenat
In reply to this post by Nicolas Burrus
Hi,

> I am also very interested by a surfel implementation in PCL. I have my
> own quick'n'dirty implementation in rgbdemo that I would be happy to
> get rid of :-)

Good, it is nice to share work :-)

> However, to be useful surfels typically require a number of associated
> data, such as the number of views in which the surfel appears, some
> confidence values, maybe some weights to compute weighted running
> average, etc. How could this be managed? Through pointers to
> associated data? Would it be possible to get a
> Surfel<AssociatedDataType>  point type? Or this would be too complex
> for further serialization?

I agree that it is the main question.

I can see the following "meta" informations for a surfel, in addition to
the minimum for visualisation:
- view histogram, currently 8x8 in published papers
- confidence, a computed value out of the view histogram
- links to scafolding data, such as the frame(s) in which the surfel has
been seen, or links to other surfels, for loop closing.
- Mike seems to be using curvature, but I have not found reference of
curvature in surfel papers as far as I remember.

It is probably nice to be able to view the confidence in rviz, so I
guess that it is why Mike has it in his Surfel structure. It might also
be nice to view the 8x8 histogram, but there it is probably better to go
for markers and ad hoc visualisation mechanisms. Most probably the
internal representation of any surfel-based mapping will differ, but at
least if we agree on the external representation it is good.

So could we add something like this to PCL?:
   struct PointSurfel;
   // Members: float x, y, z, rgb, normal[3], radius, confidence;

Should we add curvature as well?

Then, we could add a new render style to rviz for normal-oriented points
with per-point radius.

What do you think?

Have a nice day,

Stéphane

--
Dr Stéphane Magnenat
http://stephane.magnenat.net
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Radu B. Rusu
Administrator

On 02/03/2011 01:06 AM, Stéphane Magnenat wrote:

>> However, to be useful surfels typically require a number of associated
>> data, such as the number of views in which the surfel appears, some
>> confidence values, maybe some weights to compute weighted running
>> average, etc. How could this be managed? Through pointers to
>> associated data? Would it be possible to get a
>> Surfel<AssociatedDataType>   point type? Or this would be too complex
>> for further serialization?
>
> I agree that it is the main question.
>
> I can see the following "meta" informations for a surfel, in addition to
> the minimum for visualisation:
> - view histogram, currently 8x8 in published papers
> - confidence, a computed value out of the view histogram
> - links to scafolding data, such as the frame(s) in which the surfel has
> been seen, or links to other surfels, for loop closing.
> - Mike seems to be using curvature, but I have not found reference of
> curvature in surfel papers as far as I remember.
>
> It is probably nice to be able to view the confidence in rviz, so I
> guess that it is why Mike has it in his Surfel structure. It might also
> be nice to view the 8x8 histogram, but there it is probably better to go
> for markers and ad hoc visualisation mechanisms. Most probably the
> internal representation of any surfel-based mapping will differ, but at
> least if we agree on the external representation it is good.
>
> So could we add something like this to PCL?:
>     struct PointSurfel;
>     // Members: float x, y, z, rgb, normal[3], radius, confidence;
>
> Should we add curvature as well?

I think adding curvature makes sense, because it comes for free :)
xyz+pad = 4
normal+pad = 4
rgb, radius, confidence, curvature = 4

So I would arrange them like this, to get maximum efficiency for SSE:
struct PointSurfel;
// Members: float x, y, z, normal[3], curvature, rgb, radius, confidence;


If we converge on this, we could put it in before 0.9 right now, so everyone has time to test it. Stephane, can you
please commit it yourself once you're happy with the format?


> Then, we could add a new render style to rviz for normal-oriented points
> with per-point radius.
>
> What do you think?

Sounds great! Let me know when you do the rviz rendering, as we might want to support it in PCL Visualizer too.

Cheers,
Radu.
--
http://pointclouds.org
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Stéphane Magnenat
Hi,

> I think adding curvature makes sense, because it comes for free :)
> xyz+pad = 4
> normal+pad = 4
> rgb, radius, confidence, curvature = 4
>
> So I would arrange them like this, to get maximum efficiency for SSE:
> struct PointSurfel;
> // Members: float x, y, z, normal[3], curvature, rgb, radius, confidence;
>
>
> If we converge on this, we could put it in before 0.9 right now, so
> everyone has time to test it. Stephane, can you please commit it
> yourself once you're happy with the format?

Do I have commit rights to PCL? The structure you propose is fine for
me, so it is also ok if you add it.

>> Then, we could add a new render style to rviz for normal-oriented points
>> with per-point radius.
>>
>> What do you think?
>
> Sounds great! Let me know when you do the rviz rendering, as we might
> want to support it in PCL Visualizer too.

Ok, I will do it in the current of next weeks, as we are busy with other
stuff in the next days. If other people want to contribute to the rviz
implementation, they are welcome ;-)

Have a nice day,

Stéphane

--
Dr Stéphane Magnenat
http://stephane.magnenat.net
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Stéphane Magnenat
In reply to this post by Radu B. Rusu
Hi Radu,

> I think adding curvature makes sense, because it comes for free :)
> xyz+pad = 4
> normal+pad = 4
> rgb, radius, confidence, curvature = 4
>
> So I would arrange them like this, to get maximum efficiency for SSE:
> struct PointSurfel;
> // Members: float x, y, z, normal[3], curvature, rgb, radius, confidence;
>
>
> If we converge on this, we could put it in before 0.9 right now, so
> everyone has time to test it. Stephane, can you please commit it
> yourself once you're happy with the format?

While adding this to PCL, I have harvested some questions:
- Why using float for RGB (like in PointXYZRGB), as opposed to uint32_t
for RGBA (like in PointXYZRGBA)? The latter seems more generic, cleaner,
and much closer to the common practice than the former.
- How do you specify the padding? In the existing point types, I see no
padding specified when called the POINT_CLOUD_REGISTER_POINT_STRUCT macro.

Thank you, have a nice day!

Stéphane

--
Dr Stéphane Magnenat
http://stephane.magnenat.net
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Radu B. Rusu
Administrator

On 02/07/2011 09:38 AM, Stéphane Magnenat wrote:

> Hi Radu,
>
>> I think adding curvature makes sense, because it comes for free :)
>> xyz+pad = 4
>> normal+pad = 4
>> rgb, radius, confidence, curvature = 4
>>
>> So I would arrange them like this, to get maximum efficiency for SSE:
>> struct PointSurfel;
>> // Members: float x, y, z, normal[3], curvature, rgb, radius, confidence;
>>
>>
>> If we converge on this, we could put it in before 0.9 right now, so
>> everyone has time to test it. Stephane, can you please commit it
>> yourself once you're happy with the format?
>
> While adding this to PCL, I have harvested some questions:
> - Why using float for RGB (like in PointXYZRGB), as opposed to uint32_t
> for RGBA (like in PointXYZRGBA)? The latter seems more generic, cleaner,
> and much closer to the common practice than the former.

This is without a doubt the biggest hack our point clouds had to endure in ROS so far. Unfortunately it comes as a
"backwards compatible" thing that mimics the sensor_msgs/PointCloud datatypes which only handle float data.

This will be one of the first things that we need to address for PCL 2.x, once Diamondback goes out. I've set this up on
the Roadmap here: http://www.ros.org/wiki/pcl/Roadmap

> - How do you specify the padding? In the existing point types, I see no
> padding specified when called the POINT_CLOUD_REGISTER_POINT_STRUCT macro.

The best thing to do is:
  * union float[4] for x,y,z (or just use PCL_ADD_POINT4D)
  * union float[4] for normal[3]
  * curvature + rgb + radius + confidence = 16b

Cheers,
Radu.
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Stéphane Magnenat
Hi,

I've commited the PointSurfel struct. Your comments are welcome.

Have a nice day,

Stéphane

--
Dr Stéphane Magnenat
http://stephane.magnenat.net
_______________________________________________
[hidden email] / http://pointclouds.org
https://code.ros.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

ozaslan
Hi,

Is there any progress on the PCL side for RViz plugins? I would appreciate
if you can point to related libraries/plugins.

Thanks in advance.



--
Sent from: http://www.pcl-users.org/
_______________________________________________
[hidden email] / http://pointclouds.org
http://pointclouds.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: surfels and PCL

Sérgio Agostinho



On 28-01-2018 00:52, ozaslan wrote:
Hi,

Is there any progress on the PCL side for RViz plugins? I would appreciate
if you can point to related libraries/plugins.

The project handling the ROS integration of PCL seems to reside here. It's still frozen at 1.7.0 from what it seems.

From our side there's no effort or intention whatsoever to develop RViz plugins.

Cheers


_______________________________________________
[hidden email] / http://pointclouds.org
http://pointclouds.org/mailman/listinfo/pcl-users

signature.asc (836 bytes) Download Attachment