A version for Jetson Orin NX

[nsercproject]

I have found that building this project in an Orin NX-based system runs into the OpenCV v4.5.4 from the JetPack 5.1.2 not being built with the CMake option -DOPENCV_EXTRA_MODULES_PATH=<opencv_contrib>/modules. Therefore it fails at the point of including the header opencv2/cudaarithm.hpp from the file engine.h:

tensorrt-cpp-api/src/engine.h:10:10: fatal error: opencv2/cudaarithm.hpp: No such file or directory
   10 | #include <opencv2/cudaarithm.hpp> 

This is the OpenCV forum Question that I found relevant to this problem, although for OpenCv v4.5.2 I realize that I see it in v.4.5.4:
OpenCV#3427.

Can you describe the steps to build OpenCV for specific Cuda versions as required by the Jetson systems?
The Jetpack 5.1.2 (https://developer.nvidia.com/embedded/downloads/archive) installs:
Cuda 11.4.315
TensorRT 8.5.2.2
OpenCV 4.5.4 without Cuda support

[nsercproject]

I tried the script from https://forums.developer.nvidia.com/uploads/short-url/3kLERQgB4ZR0q0wgUdO9qY6lxBq.sh
It was referenced in this NVIDIA forum as the accepted answer to precisely "Compiling OpenCV on JetPack 5":
NVIDIA forums#219668/5
It asks if one wants to remove the current OpneCV (I said yes because it has no Cuda support, thus partly the error I see while building tensorrt_cpp_api on this platform. Then it installs dependencies that my Jetson did not have and then it clones the two repos: the core opencv and the opencv_contrib repos. Then it builds them. On a Jetson board, it takes well over an hour. It builds OpenCV 4.6.0. The answer says it has been proven to work on Jetpack 5.
I only needed to comment out the use of nvinfer1::DataType::kFP8 in engine.h which is not present in TensorRT v8.5.2.2 which is the one that came with JetPack 5.1.2 on my system.
It builds successfully now!

[thomaskleiven]

It would be great if there's a possibility to support the non-deprecated versions of Jetpack. From what I understand, Jetpack includes tensorrt versions from v8.5.x to v8.9.x. If the only thing causing compatibility issues is nvinfer1::DataType::kFP8, maybe we can remove it for now?

Right now, this only throws a runtime exception (L363).

@cyrusbehr what do you think?

[nsercproject]

Thomas, Thanks for taking the time to answer. I think testing would quickly tell what kind of adjustments are needed for a given Jetpack distribution. I think this would be amazing for the few of us out there that work on edge devices, writing C++ code to do high-performance inference and need to exploit the latest GPU architectures (Ampere for instance) and tensor cores by generating the engine files ourselves. Thanks for your work! I will post anything else that I notice on Jetpack 5.1.2 Sincerely, Pablo

…

________________________________ From: thomaskleiven ***@***.***> Sent: 29 May 2024 03:22 AM To: cyrusbehr/tensorrt-cpp-api ***@***.***> Cc: Pablo Adames ***@***.***>; Author ***@***.***> Subject: Re: [cyrusbehr/tensorrt-cpp-api] A version for Jetson Orin NX (Issue #58) It would be great if we could support the non-deprecated versions of Jetpack. From what I understand, Jetpack includes tensorrt versions from v8.5.x to v8.9.x. If the only thing causing compatibility issues is nvinfer1::DataType::kFP8, maybe we can remove it for now? Right now, this only throws a runtime exception (L363<https://github.com/cyrusbehr/tensorrt-cpp-api/blob/6cfef469cb6e5279db8d08067ed87f54088461c6/src/engine.h#L363>). @cyrusbehr<https://github.com/cyrusbehr> what do you think? — Reply to this email directly, view it on GitHub<#58 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/A5KQZ7B5XNNN6M5C7RMSOUDZEWM6FAVCNFSM6AAAAABHODIXTGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMZWHE2DSNRZGU>. You are receiving this because you authored the thread.

[nsercproject]

It would be great if there's a possibility to support the non-deprecated versions of Jetpack. From what I understand, Jetpack includes tensorrt versions from v8.5.x to v8.9.x. If the only thing causing compatibility issues is nvinfer1::DataType::kFP8, maybe we can remove it for now?

Right now, this only throws a runtime exception (L363).

@cyrusbehr what do you think?

To add to your comment about removing nvinfer1::DataType::kFP8. I have not experienced any need for it with the conversions I have done so far after having commented it out so I could compile. The version of the library I had to use in Jetpack 5.1.2 did not have this type defined yet. What we usually do to keep the programs compatible when we find these type of library version dependencies is to use conditional compile directives based on the Jetpack version in this case: like ```

#ifdef JETPACK_5_1_2 
...
#else
... 
// use nvinfer1::DataType::kFP8
...
#endif

The code becomes quite busy with these but at least one can support several versions of the same library in the same file.

[thomaskleiven]

I would be up for supporting non-depcrecated versions of Jetpack. Perhaps we can generalise it to check for a TENSORRT_VERSION flag. I believe nvinfer1::DataType::kFP8 is available from tensorrt>=8.6, included in Jetpack 6, which is already a production release (docs).

Something like this would default to the latest version of tensorrt while keeping support for Jetpack 5.x. In the end it's up to the owner of the repo if support for Jetson devices is desired.

In case it is, happy to put up a PR for this.

#if !defined(TENSORRT_VERSION) || TENSORRT_VERSION >= 8600
else if (tensorDataType == nvinfer1::DataType::kFP8) {
          auto msg = "Error, the model has expected output of type kFP8. This is not supported by the Engine class.";
          spdlog::error(msg);
          throw std::runtime_error(msg);
      }
#endif