OpenNi2 grabber

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenNi2 grabber

Dabox
This post was updated on .
Hi,
I am currently working with OpenNI2 and pcl and I have made a small and simple basic openni2 wrapper.
I have seen now that in the master git branch the openni2 wrapper is available since a month so i built this version and everything went well except the kinfu app. I get the following error:

CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb1_oni(boost::shared_ptr<openni_wrapper::DepthImage> const&)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:895: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb1_device(boost::shared_ptr<openni_wrapper::DepthImage> const&)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:850: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb2_device(boost::shared_ptr<openni_wrapper::Image> const&, boost::shared_ptr<openni_wrapper::DepthImage> const&, float)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:868: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb2_oni(boost::shared_ptr<openni_wrapper::Image> const&, boost::shared_ptr<openni_wrapper::DepthImage> const&, float)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:913: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::startMainLoop(bool)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:941: undefined reference to `typeinfo for pcl::ONIGrabber'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `pcl::PCDGrabber<pcl::PointXYZ>::publish(pcl::PCLPointCloud2 const&, Eigen::Matrix<float, 4, 1, 0, 4, 1> const&, Eigen::Quaternion<float, 0> const&) const':
/home/gf/Libraries/pcl/io/include/pcl/io/pcd_grabber.h:292: undefined reference to `openni_wrapper::ImageRGB24::ImageRGB24(boost::shared_ptr<xn::ImageMetaData>)'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `main':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:1211: undefined reference to `pcl::OpenNIGrabber::OpenNIGrabber(std::string const&, pcl::OpenNIGrabber::Mode const&, pcl::OpenNIGrabber::Mode const&)'
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:1217: undefined reference to `pcl::ONIGrabber::ONIGrabber(std::string const&, bool, bool)'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `main':
/usr/include/eigen3/Eigen/src/Core/util/Memory.h:301: undefined reference to `pcl::OpenNIGrabber::OpenNIGrabber(std::string const&, pcl::OpenNIGrabber::Mode const&, pcl::OpenNIGrabber::Mode const&)'



Did you finish implementing the wrapper or is it still in development?
Another important thing: I checked the grabber implementation and it seems that the x focal length and the y focal length are set to the same value, while the two values should be different to get a correct result. I got pretty good results with the following values which are the ones which also libfreenect uses:

    double focal_x_depth = 5.9421434211923247e+02;
    double focal_y_depth = 5.9104053696870778e+02;
    double center_x_depth = 3.3930780975300314e+02;
    double center_y_depth = 2.4273913761751615e+02;


Is it normal that you are using the same parameters for the two values?

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

kwaegel
Administrator
Dabox wrote
Hi,
I am currently working with OpenNI2 and pcl and I have made a small and simple basic openni2 wrapper.
I have seen now that in the master git branch the openni2 wrapper is available since a month so i built this version and everything went well except the kinfu app. I get the following error:

CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb1_oni(boost::shared_ptr<openni_wrapper::DepthImage> const&)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:895: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb1_device(boost::shared_ptr<openni_wrapper::DepthImage> const&)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:850: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb2_device(boost::shared_ptr<openni_wrapper::Image> const&, boost::shared_ptr<openni_wrapper::DepthImage> const&, float)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:868: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::source_cb2_oni(boost::shared_ptr<openni_wrapper::Image> const&, boost::shared_ptr<openni_wrapper::DepthImage> const&, float)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:913: undefined reference to `openni_wrapper::DepthImage::fillDepthImageRaw(unsigned int, unsigned int, unsigned short*, unsigned int) const'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `KinFuApp::startMainLoop(bool)':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:941: undefined reference to `typeinfo for pcl::ONIGrabber'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `pcl::PCDGrabber<pcl::PointXYZ>::publish(pcl::PCLPointCloud2 const&, Eigen::Matrix<float, 4, 1, 0, 4, 1> const&, Eigen::Quaternion<float, 0> const&) const':
/home/gf/Libraries/pcl/io/include/pcl/io/pcd_grabber.h:292: undefined reference to `openni_wrapper::ImageRGB24::ImageRGB24(boost::shared_ptr<xn::ImageMetaData>)'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `main':
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:1211: undefined reference to `pcl::OpenNIGrabber::OpenNIGrabber(std::string const&, pcl::OpenNIGrabber::Mode const&, pcl::OpenNIGrabber::Mode const&)'
/home/gf/Libraries/pcl/gpu/kinfu/tools/kinfu_app.cpp:1217: undefined reference to `pcl::ONIGrabber::ONIGrabber(std::string const&, bool, bool)'
CMakeFiles/pcl_kinfu_app.dir/kinfu_app.cpp.o: In function `main':
/usr/include/eigen3/Eigen/src/Core/util/Memory.h:301: undefined reference to `pcl::OpenNIGrabber::OpenNIGrabber(std::string const&, pcl::OpenNIGrabber::Mode const&, pcl::OpenNIGrabber::Mode const&)'



Did you finish implementing the wrapper or is it still in development?
Another important thing: I checked the grabber implementation and it seems that the x focal length and the y focal length are set to the same value, while the two values should be different to get a correct result. I got pretty good results with the following values which are the ones which also libfreenect uses:

    double focal_x_depth = 5.9421434211923247e+02;
    double focal_y_depth = 5.9104053696870778e+02;
    double center_x_depth = 3.3930780975300314e+02;
    double center_y_depth = 2.4273913761751615e+02;


Is it normal that you are using the same parameters for the two values?

Thanks
Kinfu (and most of PCL) is not using the new OpenNI 2 wrapper yet. The errors above are all related to the older OpenNI 1.x wrapper (namespace openni_wrapper). Unless you uncheck OpenNI 1, it will still try to build kinfu using it.

At some point we will need to have both grabbers return the same data types, so projects can build with either one. I already have an issue open for this.

If you want to try the new grabber, enable the CMake settings WITH_OPENNI2 and BUILD_OPENNI2, and look at the pcl_openni2_viewer test project.

The exact lens focal lengths depend on camera assembly imperfections, so the default values will rarely be exactly correct. I believe two should be the same for a perfect lens/sensor combination. You should really do manual camera calibration for best results.
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

Dabox
This post was updated on .
Strange since i have built kinfu with the precious version of pcl....Thanks anyway,
 I will take a look at it.
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

Dabox
I had a look at it but i could not understand how to get a simple cloud;
I tried with something like this :

int
main (int argc, char** argv)
{
  std::string device_id ("");
  pcl::io::OpenNI2Grabber::Mode depth_mode = pcl::io::OpenNI2Grabber::OpenNI_Default_Mode;
  pcl::io::OpenNI2Grabber::Mode image_mode = pcl::io::OpenNI2Grabber::OpenNI_Default_Mode;
  unsigned mode;

  depth_mode = pcl::io::OpenNI2Grabber::Mode (mode);
  image_mode = pcl::io::OpenNI2Grabber::Mode (mode);

  pcl::io::OpenNI2Grabber grabber (device_id, depth_mode, image_mode);
  grabber.start();
  pcl::io::openni2::Image::Ptr image;
  pcl::io::openni2::DepthImage::Ptr depth;
  grabber.imageDepthImageCallback(image, depth);

  pcl::PointCloud<pcl::PointXYZRGB>::Ptr cloud;
  cloud = grabber.convertToXYZRGBPointCloud<pcl::PointXYZRGB>(image, depth);

  return (0);
}

but i can't call convertTo..() and imageDepth..() since they are protected.
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

kwaegel
Administrator
Check out how the example program is structured. It primarily uses callbacks right now and is designed for streaming data, not requesting individual clouds.

I'll admit it's not the most intuitive interface ever, but I wrote it to match the older OpenNI 1.x grabber. I might add something more obvious like requestSingleCloud(...) or using C++11 lambda callbacks at some point.

Relevant code:
  CloudConstPtr cloud_;
...
  void
  cloud_callback (const CloudConstPtr& cloud)
  {
    FPS_CALC ("cloud callback");
    boost::mutex::scoped_lock lock (cloud_mutex_);
    cloud_ = cloud;
  }

  void
  run ()
  {
...
    boost::function<void (const CloudConstPtr&) > cloud_cb = boost::bind (&OpenNI2Viewer::cloud_callback, this, _1);
    boost::signals2::connection cloud_connection = grabber_.registerCallback (cloud_cb);

...

    while (!cloud_viewer_->wasStopped () && (image_viewer_ && !image_viewer_->wasStopped ()))
    {
...
      // See if we can get a cloud
      if (cloud_mutex_.try_lock ())
      {
        cloud_.swap (cloud);
        cloud_mutex_.unlock ();
      }
...
  // Use local "cloud" object here.
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

Dabox
Thanks,
I had a look at it and I also would suggest some simple interface for getting a cloud; I made the following:

template <typename PointT> 
class Openni2pcl{

    public:

    typename pcl::PointCloud<PointT>::Ptr get_point_cloud_openni2(openni::VideoFrameRef image, openni::VideoFrameRef depth_image, int distance, bool colored) {

    //get rgb and depth data
    const openni::RGB888Pixel* rgb_buffer = (const openni::RGB888Pixel*)image.getData();
    const uint16_t* depth_map = (const uint16_t*)depth_image.getData();

    //get image size
    int image_height = image.getHeight();
    int image_width = image.getWidth();

    int depth_height = depth_image.getHeight();
    int depth_width = depth_image.getWidth();

    //create the empty Pointcloud
    boost::shared_ptr<pcl::PointCloud<PointT>> cloud (new pcl::PointCloud<PointT>);
   
    //initialize the PointCloud height and width
    //cloud->height = std::max (image_height, depth_height);
    //cloud->width = std::max (image_width, depth_width);

    //allow infinite values for points coordinates
    cloud->is_dense = false;



    //set camera parameters for kinect
    double focal_x_depth = 5.9421434211923247e+02;
    double focal_y_depth = 5.9104053696870778e+02;
    double center_x_depth = 3.3930780975300314e+02;
    double center_y_depth = 2.4273913761751615e+02;

    float bad_point = std::numeric_limits<float>::quiet_NaN ();

    // set xyz to Nan and rgb to 0 (black)  
    if (image_width != depth_width) {
      PointT pt;
      pt.x = pt.y = pt.z = bad_point;
      pt.b = pt.g = pt.r = 0;
      pt.a = 255; // point has no color info -> alpha = max => transparent
      cloud->points.assign (cloud->points.size (), pt);
    }



    #pragma omp parallel for
    for (unsigned int y = 0; y < depth_height; ++y)
        for ( unsigned int x = 0; x < depth_width; ++x){
            PointT ptout;
            uint16_t dz = depth_map[y*depth_width + x];
            if (abs(dz) < distance){
                // project
                Eigen::Vector3d ptd((x - center_x_depth) * dz / focal_x_depth, (y - center_y_depth) * dz/focal_y_depth, dz);
                // assign output xyz
                if(ptd.x() == 0 && ptd.y() == 0 && ptd.z() == 0){
                    continue;
                }else{
                    ptout.x = ptd.x()*0.001f;
                    ptout.y = ptd.y()*0.001f;
                    ptout.z = ptd.z()*0.001f;
                }
                if(colored){
                    openni::RGB888Pixel pix = rgb_buffer[y*depth_width + x];
                    ptout.rgba = pcl::PointXYZRGB(pix.r,pix.g,pix.b).rgba; //assign color
                } else
                    ptout.rgba = pcl::PointXYZRGB(0, 0, 0).rgba;
                    #pragma omp critical
                    cloud->points.push_back(ptout); //assigns point to cloud  
            }
           
        }
/*

    cloud->sensor_origin_.setZero();
    cloud->sensor_orientation_.w () = 0.0;
    cloud->sensor_orientation_.x () = 1.0;
    cloud->sensor_orientation_.y () = 0.0;
    cloud->sensor_orientation_.z () = 0.0;

    cloud->height = 1;
    cloud->width = cloud->points.size();
*/
    return (cloud);
    }


I compiled the openni2 viewer but it is not working, I get only segmentation faults; I copied the driver in the right folder, but nothing. I will try to debug it to see where the problem lie, but for now I will continue with my simple one.
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

josuerocha
Hello Dabox, could you solve the problem with the segmentation faults? I compiled PCL + OpenNI2 + Freenect and I only get segmentation faults when I run my own code or pcl_openni2_viewer. However, the samples that came with OpenNI2 and Freenect are fine. What can I do to solve it? Is it impossible to use PCL with OpenNI2?
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

peter9606
Hi, josuerocha,

 Did you get any progress? If any, would you please share the detail guide cause I'm stucking on the pcl_openni_viewer / pcl_openni2_viewer abort too.

Regards,
Peter
Reply | Threaded
Open this post in threaded view
|

Re: OpenNi2 grabber

josuerocha
   Hi Peter, unfortunately no. Then I just switched to OpenNI and started using it to avoid problems. But I still need to acquire RGB and infrared from the camera, which are callbacks that OpenNI does not have.
   If you find a solution, please let me know.

Best regards.
__________________________________________
Josué Rocha Lima
Computer Engineering student at Federal Center for Technological Education - Timóteo, MG, Brazil
Robotics and Software Perception
josuerocha@me.com