KTX support seems to be broken (engine SIGSEGV)

[illwieckz]

KTX support seems to be broken, it's also possible that crunch produces bad KTX files.

Just to check this up, I produced some KTX textures with crunch and tried to load a map using them.

I got a SIGSEGV at texture loading time, complaining about SSSE3 stuff.

In any way, no one seems to use this image format with Dæmon engine.

It fails on the Com_Memcpy() call there (line 172):

Daemon/src/engine/renderer/tr_image_ktx.cpp

Lines 162 to 175 in 11278ed

	for(unsigned i = 1; i <= hdr->numberOfMipmapLevels; i++ ) {
	imageSize = *((uint32_t *)ptr);
	if( hdr->endianness == KTX_endianness_reverse )
	imageSize = Swap32( imageSize );
	imageSize = PAD( imageSize, 4 );
	ptr += 4;
	for(unsigned j = 0; j < hdr->numberOfFaces; j++ ) {
	int idx = i * hdr->numberOfFaces + j;
	data[ idx ] = data[ idx - 1 ] + imageSize;
	Com_Memcpy( data[ idx ], ptr, imageSize );
	ptr += imageSize;
	}
	}

What I see on console:

Warn: R_FindImageFile could not find image 'textures/test-pbr_custom_src/wall01_d' in shader 'textures/test-pbr_custom/wall01' 
Thread 1 "daemon" received signal SIGSEGV, Segmentation fault.
__memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:1

GDB backtrace:

Thread 1 (Thread 0x7ffff4b711c0 (LWP 17960)):
#0  __memcpy_ssse3 () at ../sysdeps/x86_64/multiarch/memcpy-ssse3.S:132
#1  0x0000555555839b06 in LoadKTX (name=0x55555679c9ab <va(char const*, ...)::string+3915> "textures/test-pbr_custom_src/wall01_nh.ktx", data=0x7ffffffebee0, width=0x7ffffffebea8, height=0x7ffffffebeac, numLayers=0x7ffffffebeb0, numMips=0x7ffffffebeb4, bits=0x7ffffffebe94) at Daemon/src/engine/renderer/tr_image_ktx.cpp:172
#2  0x0000555555826dbf in R_LoadImage (buffer=0x7ffffffebed0, pic=0x7ffffffebee0, width=0x7ffffffebea8, height=0x7ffffffebeac, numLayers=0x7ffffffebeb0, numMips=0x7ffffffebeb4, bits=0x7ffffffebe94) at Daemon/src/engine/renderer/tr_image.cpp:1779
#3  0x000055555582705c in R_FindImageFile (imageName=0x7fffffff4390 "textures/test-pbr_custom_src/wall01_nh", bits=131076, filterType=filterType_t::FT_DEFAULT, wrapType=...) at Daemon/src/engine/renderer/tr_image.cpp:1837
#4  0x0000555555877276 in LoadMap (stage=0x555556229220 <stages>, buffer=0x7fffffff4390 "textures/test-pbr_custom_src/wall01_nh", bundleIndex=1) at Daemon/src/engine/renderer/tr_shader.cpp:1453
#5  0x00005555558777e5 in ParseNormalMap (stage=0x555556229220 <stages>, text=0x7fffffff4c78, bundleIndex=1) at Daemon/src/engine/renderer/tr_shader.cpp:1632
#6  0x000055555587814c in ParseStage (stage=0x555556229220 <stages>, text=0x7fffffff4c78) at Daemon/src/engine/renderer/tr_shader.cpp:1826
#7  0x000055555587ca2d in ParseShader (_text=0x7fffd585db1c "materialMap textures/test-pbr_custom_src/wall01_m\n\t}\n}\n\n\n\n\n\n\n\n\n\ntextures/shared_vega/sky\n{\n\tqer_editorImage env/shared_vega_src/sky_ft\n\n\tsurfaceparm noimpact\n\tsurfaceparm nolightmap\n\tsurfaceparm sky\n\n"...) at Daemon/src/engine/renderer/tr_shader.cpp:3553
#8  0x0000555555881806 in R_FindShader (name=0x7ffff22fca00 "textures/test-pbr_custom/wall01", type=shaderType_t::SHADER_3D_STATIC, flags=RSF_DEFAULT) at Daemon/src/engine/renderer/tr_shader.cpp:5352
#9  0x00005555557fce8f in ShaderForShaderNum (shaderNum=8) at Daemon/src/engine/renderer/tr_bsp.cpp:766
#10 0x00005555557fd2e0 in ParseFace (ds=0x55555ed95e00, verts=0x55555ed939a8, surf=0x7fffd58a4ce0, indexes=0x55555edbab5c) at Daemon/src/engine/renderer/tr_bsp.cpp:854
#11 0x00005555558055f1 in R_LoadSurfaces (surfs=0x55555ed90600, verts=0x55555ed905e8, indexLump=0x55555ed905f0) at Daemon/src/engine/renderer/tr_bsp.cpp:3383
#12 0x000055555580e88d in RE_LoadWorldMap (name=0x55555d8d6bd0 "maps/test-pbr.bsp") at Daemon/src/engine/renderer/tr_bsp.cpp:6795
#13 0x00005555556f2303 in CGameVM::<lambda(const string&)>::operator()(const std::string &) const (__closure=0x7fffffff52c8, mapName="maps/test-pbr.bsp") at Daemon/src/engine/client/cl_cgame.cpp:1274
#14 0x0000555555704731 in Util::apply_impl<CGameVM::QVMSyscall(int, Util::Reader&, IPC::Channel&)::<lambda(const string&)>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, 0>(CGameVM::<lambda(const string&)> &&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > &&, Util::seq<0>) (func=..., tuple=...) at Daemon/src/common/Util.h:125
#15 0x00005555557021d7 in Util::apply<CGameVM::QVMSyscall(int, Util::Reader&, IPC::Channel&)::<lambda(const string&)>, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >(CGameVM::<lambda(const string&)> &&, std::tuple<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > &&) (func=..., tuple=...) at Daemon/src/common/Util.h:130
#16 0x00005555556fc606 in IPC::detail::HandleMsg<CGameVM::QVMSyscall(int, Util::Reader&, IPC::Channel&)::<lambda(const string&)>, IPC::Id<0, 36>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(IPC::Channel &, IPC::Message<IPC::Id<0, 36>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, Util::Reader, CGameVM::<lambda(const string&)> &&) (channel=..., reader=..., func=...) at Daemon/src/common/IPC/Channel.h:200
#17 0x00005555556f70a1 in IPC::HandleMsg<IPC::Message<IPC::Id<0, 36>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, CGameVM::QVMSyscall(int, Util::Reader&, IPC::Channel&)::<lambda(const string&)> >(IPC::Channel &, Util::Reader, CGameVM::<lambda(const string&)> &&) (channel=..., reader=..., func=...) at Daemon/src/common/IPC/Channel.h:241
#18 0x00005555556f3730 in CGameVM::QVMSyscall (this=0x555556026e20 <cgvm>, index=36, reader=..., channel=...) at Daemon/src/engine/client/cl_cgame.cpp:1272
#19 0x00005555556f1936 in CGameVM::Syscall (this=0x555556026e20 <cgvm>, id=36, reader=..., channel=...) at Daemon/src/engine/client/cl_cgame.cpp:1104
#20 0x00005555557088bf in VM::VMBase::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&)::{lambda(unsigned int, Util::Reader)#1}::operator()(unsigned int, Util::Reader) (this=0x555556026e20 <cgvm>, id=36, reader=...) at Daemon/src/engine/framework/VirtualMachine.h:141
#21 0x000055555570e8ab in IPC::detail::SendMsg<VM::VMBase::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&)::{lambda(unsigned int, Util::Reader)#1}&, IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<>, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(IPC::Channel&, VM::VMBase::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&)::{lambda(unsigned int, Util::Reader)#1}&, IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_stri--Type <RET> for more, q to quit, c to continue without paging--
ng<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&) (channel=..., messageHandler=..., args#0=@0x7fffffff56a4: 3, args#1=@0x7fffffff56a0: 0, args#2=..., args#3=...) at Daemon/src/common/IPC/Channel.h:174
#22 0x000055555570af1d in IPC::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, VM::VMBase::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&)::{lambda(unsigned int, Util::Reader)#1}, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(IPC::Channel&, VM::VMBase::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&)::{lambda(unsigned int, Util::Reader)#1}&&, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&) (channel=..., messageHandler=..., args#0=@0x7fffffff56a4: 3, args#1=@0x7fffffff56a0: 0, args#2=..., args#3=...) at Daemon/src/common/IPC/Channel.h:234
#23 0x00005555557089dd in VM::VMBase::SendMsg<IPC::SyncMessage<IPC::Message<IPC::Id<(unsigned short)0, (unsigned short)1>, int, int, glconfig_t, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul> >, IPC::Reply<> >, int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&>(int&, int&, glconfig_t&, std::array<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, 1024ul>&) (this=0x555556026e20 <cgvm>, args#0=@0x7fffffff56a4: 3, args#1=@0x7fffffff56a0: 0, args#2=..., args#3=...) at Daemon/src/engine/framework/VirtualMachine.h:139
#24 0x00005555556f14d2 in CGameVM::CGameInit (this=0x555556026e20 <cgvm>, serverMessageNum=3, clientNum=0) at Daemon/src/engine/client/cl_cgame.cpp:1021
#25 0x00005555556f0aac in CL_InitCGame () at Daemon/src/engine/client/cl_cgame.cpp:661
#26 0x00005555557409ac in CL_DownloadsComplete () at Daemon/src/engine/client/cl_download.cpp:109
#27 0x0000555555740d46 in CL_InitDownloads () at Daemon/src/engine/client/cl_download.cpp:237
#28 0x000055555575e17b in CL_ParseGamestate (msg=0x7fffffff5a20) at Daemon/src/engine/client/cl_parse.cpp:461
#29 0x000055555575e46f in CL_ParseServerMessage (msg=0x7fffffff5a20) at Daemon/src/engine/client/cl_parse.cpp:571
#30 0x000055555574e6ef in CL_PacketEvent (from=..., msg=0x7fffffff5a20) at Daemon/src/engine/client/cl_main.cpp:2428
#31 0x000055555565d172 in Com_EventLoop () at Daemon/src/engine/qcommon/common.cpp:448
#32 0x000055555565e038 in Com_Frame () at Daemon/src/engine/qcommon/common.cpp:1009
#33 0x000055555576643f in Application::ClientApplication::Frame (this=0x55555604c060 <Application::GetApp()::app>) at Daemon/src/engine/client/ClientApplication.cpp:84
#34 0x000055555590a3ce in Application::Frame () at Daemon/src/engine/framework/Application.cpp:73
#35 0x000055555594873d in main (argc=35, argv=0x7fffffffdce8) at Daemon/src/engine/framework/System.cpp:688

[illwieckz]

@dimhotepus

This is a test map with the ktx texture that is known to fail (load it as test-pbr in console):
https://dl.illwieckz.net/b/daemon/bugs/ktx-textures/map-test-pbr_0%2B20200201-194631%2Baa902da-dirty.dpk

Note: this has nothing to do with PBR, I just hijacked this map to do some texture tests.

[dimhotepus]

@illwieckz Well, KTX texture is textures/test-pbr_custom_src/wall01_d.ktx.

glType / glTypeSize is 0, so according to https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/#2.4: "for texture data which does not depend on platform endianness, including compressed texture data, glTypeSize must equal 1."
And now we support only textures which don't depend on platform endianness.

glFormat is 0, so it is compressed (https://www.khronos.org/opengles/sdk/tools/KTX/file_format_spec/#2.5).

glInternalFormat is 8D64, which is unknown for me.

glBaseInternalFormat is GL_RGB, so it is based on GL_RGB.

Unfortunately, this texture is not supported for now.

Acceptable KTX texture should have these props (https://github.com/DaemonEngine/Daemon/pull/335/files#diff-26190d20985a5f5c54f799deb59d31baR83):
glTypeSize is 1 - textures don't depend on platform endianness.
numberOfArrayElements - no arrays.
numberOfFaces - 1 or 6 cubemap faces.
numberOfMipmapLevels - less or equal to 12 mipmap levels.
pixelDepth - 0 depth.

glInternalFormat one of
GL_RGBA8,
GL_COMPRESSED_RGB_S3TC_DXT1_EXT, GL_COMPRESSED_RGBA_S3TC_DXT3_EXT, GL_COMPRESSED_RGBA_S3TC_DXT5_EXT,
GL_COMPRESSED_RED_RGTC1,
GL_COMPRESSED_RG_RGTC2 (https://github.com/DaemonEngine/Daemon/pull/335/files#diff-26190d20985a5f5c54f799deb59d31baR93).

[illwieckz]

Ok, as I said, I only produced this texture to test the feature, with the tool I had in hand, which tolds me it would be possible to have bugs. This one: https://github.com/DaemonEngine/crunch

The readme says:

Unfortunately, almost all of the tools I've found that support .KTX are fairly (to very) buggy, or are limited to only a handful of pixel formats, so there's no guarantee that the .KTX files written by crnlib can be reliably read by other tools.

It would be cool to test with textures we know they are produced the right way, but in any way if the engine does not crash it's already good As an arbitrary map downloaded from a random server must not crash the engine.

[illwieckz]

@dimhotepus on a side note related to Crunch, since you seem to have sharp eyes and good understanding of c/c++ things, would you be able to review this pull request for me?
Unity-Technologies/crunch#11

This one was submitted to upstream, but they never merge pull requests.
I would like to know if that's a fix we need, if yes I would merge it in our tree.

[dimhotepus]

@illwieckz thank you for kind words. I'll review this one after fixes for current issue are complete.

At first glance looks good. cCRNFmtDXT5 and cCRNFmtDXN_XY / cCRNFmtDXN_YX are different formats and without fix we apply cCRNFmtDXN_XY / cCRNFmtDXN_YX decompression for cCRNFmtDXT5, which is an error.