Use plane model segmentation on kinect data [screenshot inside]

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Use plane model segmentation on kinect data [screenshot inside]

Median
This post has NOT been accepted by the mailing list yet.
This post was updated on .
Hello PCL users!

I finally managed to visualize point clouds from the kinect using the OpenNI grabber tutorial, here is an example of the output:



Its easy to distinguish the hand and object from the surface on that point cloud visualizer. What I need is to segment the plane in order to detect the users hand and also objects that may be on the surface.

I tried the planar-segmentation tutorial ( http://http://pointclouds.org/documentation/tutorials/planar_segmentation.php#planar-segmentation ) but nothing seems to change in the point cloud.
Here is how i used the code (note that I started testing PCL just yesterday so I don't really know what I should expect)

void SimpleOpenNIViewer::cloud_cb_(const pcl::PointCloud<pcl::PointXYZ>::ConstPtr &cloud)
{

        if (!viewer.wasStopped())
        {
                pcl::ModelCoefficients::Ptr coefficients (new pcl::ModelCoefficients);
                pcl::PointIndices::Ptr inliers (new pcl::PointIndices);
                // Create the segmentation object
                pcl::SACSegmentation<pcl::PointXYZ> seg;
                // Optional
                seg.setOptimizeCoefficients (true);
                // Mandatory
                seg.setModelType (pcl::SACMODEL_PLANE);
                seg.setMethodType (pcl::SAC_RANSAC);
                seg.setDistanceThreshold (0.01);
                seg.setInputCloud (cloud->makeShared());
                seg.segment (*inliers, *coefficients);

                if (inliers->indices.size () == 0)
                {
                        PCL_ERROR ("Could not estimate a planar model for the given dataset.");
                        exit(0);
                }

                viewer.showCloud (cloud);

        }
}

This code runs but nothing changes in the visualization. So, what should I do to segment that surface?

Also, the framerate drops alot when i run this code.. is what I am trying to do possible at 30 fps, or we are not there just yet?

Best regards,
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

VictorLamoine
Administrator
Hello,

You are computing a planar segmentation but you don't seem to use it. You have to print the results or modify the point RGB value of outliers/inliers points to see a difference before showing the cloud.

Something like the code below show give you some info about your computation  in the console :

void SimpleOpenNIViewer::cloud_cb_(const pcl::PointCloud<pcl::PointXYZ>::ConstPtr &cloud)
{

        if (!viewer.wasStopped())
        {
                pcl::ModelCoefficients::Ptr coefficients (new pcl::ModelCoefficients);
                pcl::PointIndices::Ptr inliers (new pcl::PointIndices);
                // Create the segmentation object
                pcl::SACSegmentation<pcl::PointXYZ> seg;
                // Optional
                seg.setOptimizeCoefficients (true);
                // Mandatory
                seg.setModelType (pcl::SACMODEL_PLANE);
                seg.setMethodType (pcl::SAC_RANSAC);
                seg.setDistanceThreshold (0.01);
                seg.setInputCloud (cloud->makeShared());
                seg.segment (*inliers, *coefficients);

                if (inliers->indices.size () == 0)
                {
                        PCL_ERROR ("Could not estimate a planar model for the given dataset.");
                        exit(0);
                }

                viewer.showCloud (cloud);

                std::cerr << "Model coefficients: " << coefficients->values[0] << " "
                              << coefficients->values[1] << " "
                              << coefficients->values[2] << " "
                              << coefficients->values[3] << std::endl;


        }
}

Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

Median
This post has NOT been accepted by the mailing list yet.
Thank you for the answer. I get this output on the console
Model coefficients: 0.0371401 0.525563 0.849944 -0.974876
Model coefficients: 0.0750882 0.499831 0.862862 -0.979446
Model coefficients: 0.075247 0.499731 0.862906 -0.979454
Model coefficients: 0.0370265 0.526144 0.849589 -0.974675
Model coefficients: 0.0523925 0.515393 0.855351 -0.975484
Model coefficients: 0.0334652 0.497566 0.86678 -0.983755
...

I was hoping there was some way I could visualize this on the window, since I really don't find much meaning in this output. Any tutorial you can point me to?

About the framerate, since I am computing raw point cloud data, the framerate drops to about 5 fps. Is there anyway to compute this stuff on real time?
I mean, is it possible to perform 3d operations with PCL on real time, or is this something that is not yet possible?

Thanks for the answer.
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

VictorLamoine
Administrator
What you are doing seems to work.

Take a look at the visualization tutorial. They are showing colored point clouds with the option -r for example ! There are some explanations at the bottom

The framrate depends on your program (what are you doing with your point clouds ?) and on the capabilites of your computer (hardware)
What is your machine ?

5 fps is a good start since it's working.
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

Median
This post has NOT been accepted by the mailing list yet.
Thanks, I'll checkout that tutorial and get back to you.


I am using an oldie intel core 2 duo, it is not the best thing on the world but it has already a fair processing power.
I think that one of the reasons the framerate is so low is because I'm processing a raw point cloud, which means ~300K points per frame. There may be a way to compress this data and keep most of the information in order to speed things up
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

shiweist
Hi Median,
You can try to use voxel grid to downsample the point cloud to increase the processing speed (see http://pointclouds.org/documentation/tutorials/voxel_grid.php)

To visualize the cloud, you can color the plane and the show the cloud with color like VictorL said above.

To color the plane. Loop through the inliers, get the index and color it in the cloud.
for (int i = 0; i < inliers->indices.size(); i++)
{
  int idx = inliers->indices[i];
  cloud->points[idx].r = 255;
  cloud->points[idx].g = 0;
  cloud->points[idx].b = 0;
}

And you would probably want to have a look at the "Extracting Indices from a Pointcloud" tutorial as well (http://pointclouds.org/documentation/tutorials/extract_indices.php#extract-indices)

Good luck and have fun :)
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

Median
This post has NOT been accepted by the mailing list yet.
Yes, already had tried the voxel grid method, it downsamples a 70000 points into 5000 and the algorithm goes much faster. Not sure if I am giving up to much information or not though
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

Median
This post has NOT been accepted by the mailing list yet.
This post was updated on .
Pretty cool the plane segmentation algorithm. It paints the plane points in red and I can place objects or my hand in the surface that it knows that they dont belong to the plane:

Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

VictorLamoine
Administrator
Sharing your code would be nice for other users !
Good job.

What is the framerate now ?
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

Median
This post has NOT been accepted by the mailing list yet.
This post was updated on .
If I don't paint the surface and process PointXYZ instead of PointXYZRGBA I get max framerate. Painting the surface causes a drop on the framerate (not much though), but it is only necessary for visualizing output, so it is not a problem.


SimpleOpenNIViewer.h

#include <pcl/io/openni_grabber.h>
#include <pcl/visualization/cloud_viewer.h>
#include <pcl/point_types.h>
#include <pcl/filters/passthrough.h>
#include <pcl/ModelCoefficients.h>
#include <pcl/io/pcd_io.h>
#include <pcl/point_types.h>
#include <pcl/sample_consensus/method_types.h>
#include <pcl/sample_consensus/model_types.h>
#include <pcl/segmentation/sac_segmentation.h>
#include <pcl/filters/voxel_grid.h>

 class SimpleOpenNIViewer
 {
   public:
     SimpleOpenNIViewer ();

     void cloud_cb_ (const pcl::PointCloud<pcl::PointXYZRGBA>::ConstPtr &cloud);

     void run ();

     pcl::visualization::CloudViewer viewer;

 };

SimpleOpenNIViewer.cpp

#include "SimpleOpenNIViewer.h"

SimpleOpenNIViewer::SimpleOpenNIViewer():
viewer ("PCL OpenNI Viewer")
{
}

void SimpleOpenNIViewer::cloud_cb_(const pcl::PointCloud<pcl::PointXYZRGBA>::ConstPtr &cloud)
{
        if (!viewer.wasStopped())
        {
                pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloud_filtered (new pcl::PointCloud<pcl::PointXYZRGBA>);
                pcl::PointCloud<pcl::PointXYZRGBA>::Ptr cloud_filtered2 (new pcl::PointCloud<pcl::PointXYZRGBA>);
               
                //FILTERING PROCESS TO ELIMINATE EVERYTHING BUT THE SURFACE
                pcl::PassThrough<pcl::PointXYZRGBA> pass;
                pass.setInputCloud (cloud);
                pass.setFilterFieldName ("z");
                pass.setFilterLimits (0, 1.3);
                pass.filter (*cloud_filtered);

                pass.setInputCloud (cloud_filtered);
                pass.setFilterFieldName ("x");
                pass.setFilterLimits (-0.4, 0.4);
                //pass.setFilterLimitsNegative (true);
                pass.filter (*cloud_filtered);

                pass.setInputCloud (cloud_filtered);
                pass.setFilterFieldName ("y");
                pass.setFilterLimits (-0.15, 0.3);
                pass.filter (*cloud_filtered);

                //DOWNSAMPLING RESULTING POINT CLOUD
                pcl::VoxelGrid<pcl::PointXYZRGBA> sor;
                sor.setInputCloud (cloud_filtered);
                sor.setLeafSize (0.01f, 0.01f, 0.01f);
                sor.filter (*cloud_filtered2);
               
                //SEGMENT SURFACE
                pcl::ModelCoefficients::Ptr coefficients (new pcl::ModelCoefficients);
                pcl::PointIndices::Ptr inliers (new pcl::PointIndices);
                // Create the segmentation object
                pcl::SACSegmentation<pcl::PointXYZRGBA> seg;
                // Optional
                seg.setOptimizeCoefficients (true);

                // Mandatory
                seg.setModelType (pcl::SACMODEL_PLANE);
                seg.setMethodType (pcl::SAC_RANSAC);
                seg.setDistanceThreshold (0.01);
                seg.setInputCloud (cloud_filtered2->makeShared());
                seg.segment (*inliers, *coefficients);

                if (inliers->indices.size () == 0)
                {
                        PCL_ERROR ("Could not estimate a planar model for the given dataset.");
                  exit(0);
                }
               
                //PAINT SURFACE
                for (unsigned int i = 0; i < inliers->indices.size(); i++)
                {
                        int idx = inliers->indices[i];
                        cloud_filtered2->points[idx].r = 255;
                        cloud_filtered2->points[idx].g = 0;
                        cloud_filtered2->points[idx].b = 0;
                }

                viewer.showCloud(cloud_filtered2);
        }
}

void SimpleOpenNIViewer::run()
{
        pcl::Grabber* interface = new pcl::OpenNIGrabber();

        boost::function<void (const pcl::PointCloud<pcl::PointXYZRGBA>::ConstPtr&)> f =
        boost::bind (&SimpleOpenNIViewer::cloud_cb_, this, _1);

        interface->registerCallback (f);

        interface->start ();

        while (!viewer.wasStopped())
        {
                boost::this_thread::sleep (boost::posix_time::seconds (1));
        }

        interface->stop ();
}

main.cpp

#include "SimpleOpenNIViewer.h"

int main ()
 {
   SimpleOpenNIViewer v;
   v.run ();
   return 0;
 }
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

compeng
This post has NOT been accepted by the mailing list yet.
Hi Median,
Thanks a lot because of sharing the source code.

I wanna run this code on MSVC 2010 - win7 x64.
Can you share CmakeLists.txt and .cpp . I created three .cpp file "main.cpp .. etc" but I can not run the source code. I can run a lot of source code in my system but how can I run on MSVC 2010? Can you combine 3 cpp file into 1 cpp file?

Anybody? please help me.

Good work!
Reply | Threaded
Open this post in threaded view
|

Re: Use plane model segmentation on kinect data [screenshot inside]

kohsh
This post has NOT been accepted by the mailing list yet.
In reply to this post by Median
I am run this code but debug error occur this code

  pass.filter (*cloud_filtered);

                pass.setInputCloud (cloud_filtered);
                pass.setFilterFieldName ("y");
                pass.setFilterLimits (-0.15, 0.3);
                pass.filter (*cloud_filtered);

vector error occurred

what's wrong??

i waiting your reply

thank you