different registration result between pcl1.1 and pcl 1.7 (color error)

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

different registration result between pcl1.1 and pcl 1.7 (color error)

dinamex
This post was updated on .
Hi,

I'm registering the rgb values from an external camera to the kinect depth. I use ROS and because of that pcl1.1. Which works so far but now I want to use the BilateralUpsampling class ( only available in pcl 1.7)
So I installed the trunk version of pcl1.7 and compiled my program but as you can see the colors are not right anymore.

Does somebody know the problem? Would be great to get your help.
I also get some pragma warnings when compiling with pcl17.

Left:result with pcl 1.1
Right:result with pcl 1.7
result with pcl 1.1

result with pcl17
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

aichim
Administrator
Hi,

There have been quite some changes in the internals of PCL since 1.1 (that was years ago :-) ), including some tweaks on the file format. Please try capturing data with the new PCL, and see if you get the same issues.

If you have some very precious datasets recorded a long time ago, we could try and find a solution for you.

Cheers,
Alex
---
http://www.openperception.org



On Feb 24, 2013, at 2:15 PM, dinamex <[hidden email]> wrote:

> Hi,
>
> I'm registering the rgb values from an external camera to the kinect depth.
> I use ROS and because of that pcl1.1. Which works so far but now I want to
> use the BilateralUpsampling class ( only available in pcl 1.7)
> So I installed the trunk version of pcl1.7 and compiled my program but as
> you can see the colors are not right anymore.
>
> Does somebody know the problem? Would be great to get your help.
>
>
> <http://www.pcl-users.org/file/n4026334/Screenshot_from_2013-02-24_17_09_19.png>
>
> <http://www.pcl-users.org/file/n4026334/Screenshot_from_2013-02-24_17_07_57.png>
>
>
>
> --
> View this message in context: http://www.pcl-users.org/different-registration-result-between-pcl1-1-and-pcl-1-7-color-error-tp4026334.html
> Sent from the Point Cloud Library (PCL) Users mailing list mailing list archive at Nabble.com.
> _______________________________________________
> [hidden email] / http://pointclouds.org
> http://pointclouds.org/mailman/listinfo/pcl-users

_______________________________________________
[hidden email] / http://pointclouds.org
http://pointclouds.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

dinamex
what do you mean by file format? I just have a jpg image and a depth image and reconstruct a cloud from these two images... these images are captured with my own grabber (based on the openni framework)


Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

aichim
Administrator
Oh, I seeā€¦ Then what modules are you using for that?


Cheers,
Alex
---
http://www.openperception.org



On Feb 24, 2013, at 2:50 PM, dinamex <[hidden email]> wrote:

> what do you mean by file format? I just have a jpg image and a depth image
> and reconstruct a cloud from these two images... these images are captured
> with my own grabber (based on the openni framework)
>
>
>
>
>
>
> --
> View this message in context: http://www.pcl-users.org/different-registration-result-between-pcl1-1-and-pcl-1-7-color-error-tp4026334p4026336.html
> Sent from the Point Cloud Library (PCL) Users mailing list mailing list archive at Nabble.com.
> _______________________________________________
> [hidden email] / http://pointclouds.org
> http://pointclouds.org/mailman/listinfo/pcl-users

_______________________________________________
[hidden email] / http://pointclouds.org
http://pointclouds.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

dinamex
you mean the openni modules? where should be the difference the when i just use a jpg and a depth image?



here is a minimal program which loads the depth image and rgb image -> computes the 3 position of the point -> project points to rgb image -> get the color for the 3D point and save it to a pcd file.
Nothing special, but doesn't work with pcl1.7 but does for pcl1.1

#include "pcl/io/pcd_io.h"
#include "pcl/point_types.h"
#include "pcl/common/transforms.h"
//#include "pcl/surface/bilateral_upsampling.h"
#include "opencv/cv.h"
#include "opencv/highgui.h"
#include "std_msgs/String.h"

using namespace std;

int main (int argc, char **argv)
{

        //pcl::BilateralUpsampling<pcl::PointXYZRGB,pcl::PointXYZRGB> BUcloud;

        cv::Mat distCoeffs(4,1,cv::DataType<float>::type);
         distCoeffs.at<float>(0) = 0;
         distCoeffs.at<float>(1) = 0;
         distCoeffs.at<float>(2) = 0;
         distCoeffs.at<float>(3) = 0;

        //rodrigues
        cv::Mat rvec(3,1,cv::DataType<double>::type);
        rvec.at<double>(0,0) = 0.034430832;
        rvec.at<double>(1,0) = 0.012224533;
        rvec.at<double>(2,0) = 0.008440651;

        cv::Mat T(3,1,cv::DataType<double>::type);
        T.at<double>(0,0) = -2.713068269045391e-002;
        T.at<double>(1,0) = -2.469161072738409e-002;
        T.at<double>(2,0) = -2.198079469084248e-003;
        //T.at<float>(3,0) = 1;

        cv::Mat K(3,3,cv::DataType<float>::type);
        K.at<float>(0,0) = 1390.107788;
        K.at<float>(1,0) = 0;
        K.at<float>(2,0) = 0;

        K.at<float>(0,1) = 0;
        K.at<float>(1,1) = 1408.857522;
        K.at<float>(2,1) = 0;

        K.at<float>(0,2) = 964.889818;
        K.at<float>(1,2) = 541.379008;
        K.at<float>(2,2) = 1;

        for(int i=0 ; i<atoi(argv[1]) ; i++){


                cv::Mat depthImage = cv::imread("depth.png", CV_LOAD_IMAGE_UNCHANGED);
                cv::Mat rgbImage = cv::imread("rgb.jpg", CV_LOAD_IMAGE_UNCHANGED);

                // calibration information for 3d reconstruction of the cloud
                Eigen::Matrix3f rgbIntrinsicMatrix;
                rgbIntrinsicMatrix << 264.708887, 0, 154.567938, 0, 268.613938, 125.519443, 0, 0, 1;

                if(depthImage.empty() || rgbImage.empty()){
                        cout<<"could not read image files"<<endl;
                        return 0;
                }

                float rgbFocalInvertedX = 1.0/rgbIntrinsicMatrix(0,0); // 1/fx
                float rgbFocalInvertedY = 1.0/rgbIntrinsicMatrix(1,1); // 1/fy

                pcl::PointXYZRGB pclPoint;
                pcl::PointCloud<pcl::PointXYZRGB>  outputPointcloud;

                std::vector<cv::Point3f> objectPoints;
                std::vector<cv::Point2f> projectedPoints;
                cv::Point3f newPoint;


                for (int y=0;y<depthImage.rows;y++)
                {
                        for (int x=0;x<depthImage.cols;x++)
                        {
                                unsigned short depthValue = depthImage.at<unsigned short>(y,x);

                                if (depthValue == depthValue && depthValue != 0)                // if depthValue is not NaN
                                {
                                        // Find 3D position respect to rgb frame:
                                        newPoint.z = float(depthValue)*0.001f; //from mm to m
                                        newPoint.x = (x - rgbIntrinsicMatrix(0,2)) * newPoint.z * rgbFocalInvertedX;
                                        newPoint.y = (y - rgbIntrinsicMatrix(1,2)) * newPoint.z * rgbFocalInvertedY;
                                        objectPoints.push_back(newPoint);

                                }
                                else
                                {
                                        newPoint.z = std::numeric_limits<float>::quiet_NaN();
                                        newPoint.x = std::numeric_limits<float>::quiet_NaN();
                                        newPoint.y = std::numeric_limits<float>::quiet_NaN();
                                        objectPoints.push_back(newPoint);
                                }
                        }
                }
        //project points on external camera
        cv::projectPoints(objectPoints, rvec, T, K, distCoeffs, projectedPoints);

        for(unsigned int i = 0; i < projectedPoints.size(); ++i){

                //get rgb value of the point and save to pcl pointXYZRGB
                //std::cout << " projectedPoints.size() " << projectedPoints.size()<< std::endl;

                if(projectedPoints[i].y >= 0 &&projectedPoints[i].x >= 0 &&projectedPoints[i].y <=1080 &&projectedPoints[i].x <=1920 && projectedPoints[i].x == projectedPoints[i].x && projectedPoints[i].y == projectedPoints[i].y){
                    pclPoint.z = objectPoints[i].z;
                    pclPoint.x = objectPoints[i].x;
                    pclPoint.y = objectPoints[i].y;
                    pclPoint.r = rgbImage.at<cv::Vec3b>(projectedPoints[i].y, projectedPoints[i].x)[2];
                    pclPoint.g = rgbImage.at<cv::Vec3b>(projectedPoints[i].y, projectedPoints[i].x)[1];
                    pclPoint.b = rgbImage.at<cv::Vec3b>(projectedPoints[i].y, projectedPoints[i].x)[0];
                    outputPointcloud.push_back(pclPoint);

                    }
                    else
                        {

                    pclPoint.z = std::numeric_limits<float>::quiet_NaN();
                    pclPoint.x = std::numeric_limits<float>::quiet_NaN();
                    pclPoint.y = std::numeric_limits<float>::quiet_NaN();
                    pclPoint.r = std::numeric_limits<unsigned char>::quiet_NaN();
                    pclPoint.g = std::numeric_limits<unsigned char>::quiet_NaN();
                    pclPoint.b = std::numeric_limits<unsigned char>::quiet_NaN();
                                outputPointcloud.push_back(pclPoint);
                        }
        }


                //save pointcloud to disk
                pcl::PCDWriter writer;
                cout<<"Saving cloud..."<<endl;
                std::string filename = base_folder+"/pointclouds/upscaled_pointcloud_"+cloudIdxString+".pcd";
                writer.write (filename.c_str(), outputPointcloud, false);
        }
}
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

aichim
Administrator
Try writing with the file as binary.

Cheers,
Alex
---
http://www.openperception.org



On Feb 24, 2013, at 3:21 PM, dinamex <[hidden email]> wrote:

> you mean the openni modules? where should be the difference the when i just
> use a jpg and a depth image?
>
>
>
> here is a minimal program which loads the depth image and rgb image ->
> computes the 3 position of the point -> project points to rgb image -> get
> the color for the 3D point and save it to a pcd file.
> Nothing special, but doesn't work with pcl1.7 but does for pcl1.1
>
> #include "pcl/io/pcd_io.h"
> #include "pcl/point_types.h"
> #include "pcl/common/transforms.h"
> //#include "pcl/surface/bilateral_upsampling.h"
> #include "opencv/cv.h"
> #include "opencv/highgui.h"
> #include "std_msgs/String.h"
>
> using namespace std;
>
> int main (int argc, char **argv)
> {
>
> //pcl::BilateralUpsampling<pcl::PointXYZRGB,pcl::PointXYZRGB> BUcloud;
>
> cv::Mat distCoeffs(4,1,cv::DataType<float>::type);
> distCoeffs.at<float>(0) = 0;
> distCoeffs.at<float>(1) = 0;
> distCoeffs.at<float>(2) = 0;
> distCoeffs.at<float>(3) = 0;
>
> //rodrigues
> cv::Mat rvec(3,1,cv::DataType<double>::type);
> rvec.at<double>(0,0) = 0.034430832;
> rvec.at<double>(1,0) = 0.012224533;
> rvec.at<double>(2,0) = 0.008440651;
>
> cv::Mat T(3,1,cv::DataType<double>::type);
> T.at<double>(0,0) = -2.713068269045391e-002;
> T.at<double>(1,0) = -2.469161072738409e-002;
> T.at<double>(2,0) = -2.198079469084248e-003;
> //T.at<float>(3,0) = 1;
>
> cv::Mat K(3,3,cv::DataType<float>::type);
> K.at<float>(0,0) = 1390.107788;
> K.at<float>(1,0) = 0;
> K.at<float>(2,0) = 0;
>
> K.at<float>(0,1) = 0;
> K.at<float>(1,1) = 1408.857522;
> K.at<float>(2,1) = 0;
>
> K.at<float>(0,2) = 964.889818;
> K.at<float>(1,2) = 541.379008;
> K.at<float>(2,2) = 1;
>
> for(int i=0 ; i<atoi(argv[1]) ; i++){
>
>
> cv::Mat depthImage = cv::imread(&quot;depth.png&quot;,
> CV_LOAD_IMAGE_UNCHANGED);
> cv::Mat rgbImage = cv::imread(&quot;rgb.jpg&quot;,
> CV_LOAD_IMAGE_UNCHANGED);
>
> // calibration information for 3d reconstruction of the cloud
> Eigen::Matrix3f rgbIntrinsicMatrix;
> rgbIntrinsicMatrix &lt;&lt; 264.708887, 0, 154.567938, 0, 268.613938,
> 125.519443, 0, 0, 1;
>
> if(depthImage.empty() || rgbImage.empty()){
> cout&lt;&lt;&quot;could not read image files&quot;&lt;&lt;endl;
> return 0;
> }
>
> float rgbFocalInvertedX = 1.0/rgbIntrinsicMatrix(0,0); // 1/fx
> float rgbFocalInvertedY = 1.0/rgbIntrinsicMatrix(1,1); // 1/fy
>
> pcl::PointXYZRGB pclPoint;
> pcl::PointCloud&lt;pcl::PointXYZRGB>  outputPointcloud;
>
> std::vector<cv::Point3f> objectPoints;
> std::vector<cv::Point2f> projectedPoints;
> cv::Point3f newPoint;
>
>
> for (int y=0;y<depthImage.rows;y++)
> {
> for (int x=0;x&lt;depthImage.cols;x++)
> {
> unsigned short depthValue = depthImage.at&lt;unsigned short>(y,x);
>
> if (depthValue == depthValue && depthValue != 0)                // if
> depthValue is not NaN
> {
> // Find 3D position respect to rgb frame:
> newPoint.z = float(depthValue)*0.001f; //from mm to m
> newPoint.x = (x - rgbIntrinsicMatrix(0,2)) * newPoint.z *
> rgbFocalInvertedX;
> newPoint.y = (y - rgbIntrinsicMatrix(1,2)) * newPoint.z *
> rgbFocalInvertedY;
> objectPoints.push_back(newPoint);
>
> }
> else
> {
> newPoint.z = std::numeric_limits<float>::quiet_NaN();
> newPoint.x = std::numeric_limits<float>::quiet_NaN();
> newPoint.y = std::numeric_limits<float>::quiet_NaN();
> objectPoints.push_back(newPoint);
> }
> }
> }
>        //project points on external camera
> cv::projectPoints(objectPoints, rvec, T, K, distCoeffs, projectedPoints);
>
> for(unsigned int i = 0; i < projectedPoints.size(); ++i){
>
> //get rgb value of the point and save to pcl pointXYZRGB
> //std::cout << " projectedPoints.size() " << projectedPoints.size()<<
> std::endl;
>
> if(projectedPoints[i].y >= 0 &&projectedPoints[i].x >= 0
> &&projectedPoints[i].y <=1080 &&projectedPoints[i].x <=1920 &&
> projectedPoints[i].x == projectedPoints[i].x && projectedPoints[i].y ==
> projectedPoints[i].y){
>     pclPoint.z = objectPoints[i].z;
>     pclPoint.x = objectPoints[i].x;
>     pclPoint.y = objectPoints[i].y;
>     pclPoint.r = rgbImage.at<cv::Vec3b>(projectedPoints[i].y,
> projectedPoints[i].x)[2];
>     pclPoint.g = rgbImage.at<cv::Vec3b>(projectedPoints[i].y,
> projectedPoints[i].x)[1];
>     pclPoint.b = rgbImage.at<cv::Vec3b>(projectedPoints[i].y,
> projectedPoints[i].x)[0];
>     outputPointcloud.push_back(pclPoint);
>
>    }
>    else
> {
>
>     pclPoint.z = std::numeric_limits<float>::quiet_NaN();
>     pclPoint.x = std::numeric_limits<float>::quiet_NaN();
>     pclPoint.y = std::numeric_limits<float>::quiet_NaN();
>     pclPoint.r = std::numeric_limits<unsigned char>::quiet_NaN();
>     pclPoint.g = std::numeric_limits<unsigned char>::quiet_NaN();
>     pclPoint.b = std::numeric_limits<unsigned char>::quiet_NaN();
> outputPointcloud.push_back(pclPoint);
> }
> }
>
>
> //save pointcloud to disk
> pcl::PCDWriter writer;
> cout<<"Saving cloud..."<<endl;
> std::string filename =
> base_folder+"/pointclouds/upscaled_pointcloud_"+cloudIdxString+".pcd";
> writer.write (filename.c_str(), outputPointcloud, false);
> }
> }
>
>
>
>
> --
> View this message in context: http://www.pcl-users.org/different-registration-result-between-pcl1-1-and-pcl-1-7-color-error-tp4026334p4026338.html
> Sent from the Point Cloud Library (PCL) Users mailing list mailing list archive at Nabble.com.
> _______________________________________________
> [hidden email] / http://pointclouds.org
> http://pointclouds.org/mailman/listinfo/pcl-users

_______________________________________________
[hidden email] / http://pointclouds.org
http://pointclouds.org/mailman/listinfo/pcl-users
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

dinamex
In reply to this post by dinamex
same result... :(
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

dinamex
Is maybe the pragma warning the problem?

In file included from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/common/impl/io.hpp:45:0,
                 from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/common/io.h:488,
                 from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/io/file_io.h:42,
                 from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/io/pcd_io.h:44,
                 from /home/ronny/ros_workspace/reconstruction/src/RegisterWebcam2PCD.cpp:1:
/home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/point_types.h:56:0: warning: ignoring #pragma warning  [-Wunknown-pragmas]
In file included from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/common/impl/io.hpp:45:0,
                 from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/common/io.h:488,
                 from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/io/file_io.h:42,
                 from /home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/io/pcd_io.h:44,
                 from /home/ronny/ros_workspace/reconstruction/src/RegisterWebcam2PCD.cpp:1:
/home/ronny/ros_workspace/perception_pcl_unstable/pcl17/include/pcl-1.7/pcl17/point_types.h:646:0: warning: ignoring #pragma warning  [-Wunknown-pragmas]
Reply | Threaded
Open this post in threaded view
|

Re: different registration result between pcl1.1 and pcl 1.7 (color error)

dinamex
I still couldn't find a solution for this problem. Does somebody have any idea?