package import error

[kiminh]

./pixiw run build

when i build tensornet and want to import the package, such error occurs. which step should I do to fix it? @gzm55

# /data1/tensornet/.pixi/envs/default/bin/python3.8
Python 3.8.19 | packaged by conda-forge | (default, Mar 20 2024, 12:47:35) 
[GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensornet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data1/tensornet/tensornet/__init__.py", line 15, in <module>
    from . import core as core
  File "/data1/tensornet/tensornet/core/__init__.py", line 16, in <module>
    from tensornet.core._pywrap_tn import init as init
ModuleNotFoundError: No module named 'tensornet.core._pywrap_tn'
>>> import sys
>>> sys.path.append('/data1/tensornet') 
>>> import tensornet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data1/tensornet/tensornet/__init__.py", line 15, in <module>
    from . import core as core
  File "/data1/tensornet/tensornet/core/__init__.py", line 16, in <module>
    from tensornet.core._pywrap_tn import init as init
ModuleNotFoundError: No module named 'tensornet.core._pywrap_tn'

[gzm55]

now source directory is not an editable package. to build an test locally, run ./pixiw run build.
Or you can build the whl package, run ./pixiw run create-wheel, and install the whl file under dist/ folder into a separated virtual env.

[thundercatli]

@kiminh For local test, you can copy build/debug/_pywrap_tn.cpython-38-x86_64-linux-gnu.so and build/debug/libtensornet.so to project root tensornet/core , then you can start python and import. But it's better to follow @gzm55 's instructions

[kiminh]

thanks for your suggestion

[kiminh]

one more suggestion, build tensornet depends on mpi installation. It should be included in readme.

[gzm55]

one more suggestion, build tensornet depends on mpi installation. It should be included in readme.

the conda mpi packaged is installed into the default environment by pixi before executing cmake.

[kiminh]

finnaly I install tensornet whl successfully with env python 3.7.4 and tensorflow==2.2.3

-rw-r--r--  1 root root 3053901 May 16 15:46 qihoo_tensornet-0.102.0.dev0-cp35-abi3-manylinux2010_x86_64.whl
(py37_env) [root@tt /data1/tensornet/dist]
# pip install qihoo_tensornet-0.102.0.dev0-cp35-abi3-manylinux2010_x86_64.whl 

Successfully installed numpy-1.18.5 qihoo-tensornet-0.102.0.dev0 tensorflow-cpu-2.2.3

however, there still has error to import this package. I will try [thundercatli]'s solution

(py37_env) 
Python 3.7.4 (default, May 19 2025, 12:53:04) 
[GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import tensornet
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data1/py37_env/lib/python3.7/site-packages/tensornet/__init__.py", line 15, in <module>
    from . import core as core
  File "/data1/py37_env/lib/python3.7/site-packages/tensornet/core/__init__.py", line 16, in <module>
    from tensornet.core._pywrap_tn import init as init
ImportError: _pywrap_tensorflow_internal.so: cannot open shared object file: No such file or directory

[kiminh]

I tried @thundercatli solution and met such error

# python
Python 3.7.4 (default, May 19 2025, 12:53:04) 
[GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.append('/data1/tensornet') 
>>> import tensornet
[libprotobuf ERROR external/com_google_protobuf/src/google/protobuf/descriptor_database.cc:118] File already exists in database: tensorflow/core/protobuf/master.proto
[libprotobuf FATAL external/com_google_protobuf/src/google/protobuf/descriptor.cc:1367] CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: GeneratedDatabase()->Add(encoded_file_descriptor, size): 
Aborted

Then I downgrade protobuf version to protobuf==3.19.4 / protobuf==3.8.0 (tried both) and such error still exists.

How to fix it? @gzm55

[kiminh]

is this version qihoo-tensornet-0.102.0.dev0. too old?

and which python version is preferred? py3.7 or py3.8

[gzm55]

is this version qihoo-tensornet-0.102.0.dev0. too old?

qihoo-tensornet package is updated.

The libprotobuf of compile-time and run-time should both align to tensorflow (tf 2.2.3 use pb 3.8).

[kiminh]

is there a separated libprotobuf.so on runtime library searching paths

I use virtual env and confirm that there is only one version of protobuf in pip list

[gzm55]

and which python version is preferred? py3.7 or py3.8

when build .whl with ./pixiw run create-wheel command, both py 3.7 and py 3.8 should work.

[kiminh]

is this version qihoo-tensornet-0.102.0.dev0. too old?

qihoo-tensornet package is updated.

The libprotobuf of compile-time and run-time should both align to tensorflow (tf 2.2.3 use pb 3.8).

I‘m trying to rebuild the package in my virtual env py37

[gzm55]

when install tn, try use a conda env, not a python virtual env.

[gzm55]

and it's better to post the conda environment.yml to reproduce the issue.

[kiminh]

is this version qihoo-tensornet-0.102.0.dev0. too old?

and which python version is preferred? py3.7 or py3.8

I‘m confused about the builded dist package version. I run ./pixiw run create-wheel in master branch and get the following whl. Is this version correct? @gzm55

[/data1/tensornet_bak/dist] 
# ll
total 2988
drwxr-xr-x  2 root root      77 May 19 15:36 ./
drwxr-xr-x 17 root root    4096 May 16 15:40 ../
-rw-r--r--  1 root root 3053878 May 19 15:36 qihoo_tensornet-0.102.0.dev0-cp35-abi3-manylinux2010_x86_64.whl

[kiminh]

finally I solved it. thanks all you guys @gzm55 @thundercatli
let me summarize the step later.

>>> import tensorflow as tf
>>> import sys
>>> sys.path.append('/data1/tensornet_bak') 
>>> import tensornet
>>> print(tensornet.__version__)
0.102.0.dev

and the demo wide_deep example is tested ok.

[kiminh]

# 安装conda
conda env create -f config/tn_build.yaml
conda activate tn_build

cd tensornet
# 编译
./pixiw run build
# 涉及到mpi权限，如下方式定义两个环境变量来允许root下也可以用mpirun
export OMPI_ALLOW_RUN_AS_ROOT=1
export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1

# 打包
./pixiw run create-wheel

# level_db相关
sudo apt-get install libleveldb-dev
export LD_LIBRARY_PATH=/data/tensornet/.pixi/envs/default/lib:$LD_LIBRARY_PATH  # must

pip install dist/qihoo_tensornet-0.102.0.dev0-cp35-abi3-manylinux2010_x86_64.whl


### 测试代码
import sys
sys.path.append('/data/tensornet')  #must
import tensornet as tn

[gzm55]

level_db相关

sudo apt-get install libleveldb-dev
export LD_LIBRARY_PATH=/data/tensornet/.pixi/envs/default/lib:$LD_LIBRARY_PATH

micromamba install leveldb==1.18=h00d4201_1
也可以安装这个版本的 levedb conda 包. cmake 打包时 leveldb 依赖了动态库, 编译时 _GLIBCXX_USE_CXX11_ABI=0, 这个 leveldb 是最后按照 abi=0 打包的版本.