2023-01-07 15:57:01,210 - speechbrain.core - INFO - Beginning experiment! 2023-01-07 15:57:01,210 - speechbrain.core - INFO - Experiment folder: partly_frozen_splitted_wavlm 2023-01-07 15:57:01,912 - speechbrain.utils.superpowers - DEBUG - abkhazia==1.0 absl-py==0.11.0 aiohttp==3.8.0 aiosignal==1.2.0 alabaster==0.7.12 alembic==1.7.4 altgraph==0.17 antlr4-python3-runtime==4.8 appdirs==1.4.4 argcomplete==1.12.2 argon2-cffi==20.1.0 asgiref==3.6.0 astunparse==1.6.3 async-generator==1.10 async-timeout==4.0.0 attrdict==2.0.1 attrs==20.3.0 audeer==1.16.0 audformat==0.11.5 audinterface==0.7.0 audiofile==1.0.0 audiomentations==0.25.0 audioread==2.1.9 audobject==0.4.14 audresample==0.1.6 -e git+https://github.com/facebookresearch/WavAugment.git@54afcdb00ccc852c2f030f239f8532c9562b550e#egg=augment autopage==0.4.0 Babel==2.9.0 backcall==0.2.0 beautifulsoup4==4.10.0 black==19.10b0 bleach==3.3.0 boto3==1.20.2 botocore==1.23.2 braceexpand==0.1.7 cachetools==4.2.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.14.3 cfgv==3.2.0 chardet==3.0.4 charset-normalizer==2.0.7 click==7.1.2 cliff==3.9.0 clldutils==3.5.4 cmaes==0.8.2 cmake==3.18.4.post1 cmd2==2.2.0 colorama==0.4.4 colorlog==4.6.2 configparser==5.1.0 cryptography==38.0.4 csvw==1.8.1 cycler==0.10.0 Cython==0.29.21 dataclasses==0.6 datasets==1.5.0 decorator==4.4.2 deepspeech==0.9.1 defusedxml==0.7.1 denoiser==0.1.5 dill==0.3.3 Distance==0.1.3 distlib==0.3.1 Django==3.2.16 django-auditlog==2.2.1 django-filter==22.1 django-js-asset==1.2.2 django-mptt==0.14.0 djangorestframework==3.14.0 docker-pycreds==0.4.0 docopt==0.6.2 docutils==0.16 drf-excel==2.2.0 drf-flex-fields==1.0.0 drf-renderer-xlsx==0.4.1 easyocr==1.2.1 editdistance==0.6.0 emoji==2.2.0 entrypoints==0.3 et-xmlfile==1.1.0 exceptiongroup==1.1.0 farasapy==0.0.14 fasttext==0.9.2 ffmpeg-python==0.2.0 filelock==3.0.12 flake8==3.7.9 flatbuffers==1.12 frozendict==2.0.7 frozenlist==1.2.0 fsspec==2021.11.0 future==0.18.2 g2p-en==2.1.0 gast==0.3.3 gdown==4.2.0 gensim==4.0.1 gitdb==4.0.9 GitPython==3.1.24 google-auth==1.24.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 greenlet==1.1.2 grpcio==1.32.0 h5features==1.3.2 h5py==2.10.0 htk-io==0.5 huggingface-hub==0.9.1 hydra-colorlog==0.1.4 hydra-core==0.11.3 HyperPyYAML==1.1.0 hypothesis==6.61.2 identify==1.5.10 idna==2.10 imageio==2.9.0 imagesize==1.2.0 importlib-metadata==4.8.1 importlib-resources==5.2.2 inflect==5.3.0 ipadic==1.0.0 ipykernel==5.3.4 ipython==7.19.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 iso-639==0.4.5 isodate==0.6.0 isort==4.3.21 jedi==0.17.2 jieba==0.42.1 Jinja2==2.11.2 jiwer==2.2.0 jmespath==0.10.0 joblib==0.17.0 jsonschema==3.2.0 julius==0.2.7 jupyter-client==6.1.7 jupyter-core==4.7.0 jupyterlab-pygments==0.1.2 jupyterlab-widgets==1.0.0 kaitaistruct==0.9 kaldi-io==0.9.4 kaldi-python-io==1.2.2 kaldiio==2.17.2 kenlm @ https://github.com/kpu/kenlm/archive/master.zip Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 lang-trans==0.6.0 latexcodec==2.0.1 ldap3==2.9.1 librosa==0.9.0 llvmlite==0.35.0 lxml==4.9.0 Mako==1.1.5 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.14.0 matplotlib==3.3.3 mccabe==0.6.1 mcd==0.4 mecab-python3==1.0.3 megatron-lm==2.2.0 mido==1.2.10 mistune==0.8.4 more-itertools==8.6.0 mpmath==1.2.1 multidict==5.2.0 multiprocess==0.70.11.1 nbclient==0.5.3 nbconvert==6.0.7 nbformat==5.1.3 NEMO==4.3.2 nemo-toolkit==1.4.0 nest-asyncio==1.5.1 networkx==2.5 nltk==3.5 nodeenv==1.5.0 notebook==6.3.0 numba==0.52.0 numpy==1.19.4 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 oauthlib==3.1.0 omegaconf==1.4.1 onnx==1.10.2 OpenCC==1.1.2 opencv-python==4.4.0.46 openpyxl==3.0.9 opensmile==2.2.0 opt-einsum==3.3.0 optuna==2.10.0 oyaml==1.0 packaging==22.0 pandas==1.2.5 pandocfilters==1.4.3 pangu==4.0.6.1 parameterized==0.8.1 parso==0.7.1 pathspec==0.8.1 pathtools==0.1.2 pbr==5.6.0 pefile==2019.4.18 pescador==2.1.0 pesq==0.0.3 pexpect==4.8.0 phonemizer==2.2.1 pickleshare==0.7.5 Pillow==9.3.0 pip-api==0.0.23 pipreqs==0.4.11 pluggy==0.13.1 pooch==1.3.0 portalocker==2.3.2 pre-commit==2.9.0 pretty-midi==0.2.9 prettytable==2.2.1 progressbar2==3.53.1 prometheus-client==0.10.1 promise==2.3 prompt-toolkit==3.0.8 protobuf==3.14.0 psutil==5.6.6 ptyprocess==0.6.0 py==1.9.0 py-espeak-ng==0.1.8 pyannote.audio==1.1.1 pyannote.core==4.3 pyannote.database==4.1.1 pyannote.metrics==3.1 pyannote.pipeline==1.5.2 PyArabic==0.6.15 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.8.1 pybtex==0.24.0 pybtex-docutils==1.0.1 pycodestyle==2.5.0 pycparser==2.20 pyctcdecode==0.4.0 pyDeprecate==0.3.1 pydub==0.25.1 pyflakes==2.1.1 Pygments==2.7.2 pygtrie==2.5.0 pymodbus==2.5.3 pyparsing==2.4.7 pyperclip==1.8.2 pypinyin==0.43.0 pyrsistent==0.17.3 pyserial==3.5 PySocks==1.7.1 pystoi==0.3.3 pytest==5.4.1 pytest-runner==5.3.1 python-bidi==0.4.2 python-crfsuite==0.9.7 python-dateutil==2.8.2 python-Levenshtein==0.12.2 python-utils==2.4.0 pytorch-lightning==1.4.9 pytube==11.0.1 pytz==2022.6 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==20.0.0 rapidfuzz==1.8.2 regex==2020.11.13 requests==2.28.1 requests-oauthlib==1.3.0 resampy==0.2.2 rfc3986==1.4.0 rsa==4.7 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 s3m==1.1.0 s3transfer==0.5.0 sacrebleu==2.0.0 sacremoses==0.0.44 scikit-image==0.18.1 scikit-learn==0.23.2 scipy==1.5.4 -e git+https://github.com/sanghack81/SDCIT@00d060dde733fde9345154a494f81e97fb395ca7#egg=SDCIT seaborn==0.11.1 segments==2.1.3 Send2Trash==1.5.0 sentencepiece==0.1.94 sentry-sdk==1.4.3 shellingham==1.4.0 shortuuid==1.0.7 SIDEKIT==1.3.8.5.2 simplejson==3.17.5 six==1.15.0 smart-open==5.0.0 smmap==5.0.0 snowballstemmer==2.0.0 sortedcollections==2.1.0 sortedcontainers==2.4.0 sounddevice==0.4.5 SoundFile==0.10.3.post1 soupsieve==2.3 sox==1.4.1 sparsemax==0.1.9 speechbrain==0.5.13 sphfile==1.0.3 Sphinx==3.3.1 sphinx-rtd-theme==0.4.3 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-bibtex==2.4.1 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 SQLAlchemy==1.4.25 sqlparse==0.4.2 stanza==1.4.2 stevedore==3.4.0 subprocess32==3.5.4 sympy==1.9 tabulate==0.8.9 tensorboard==2.4.0 tensorboard-plugin-wit==1.7.0 tensorflow==2.4.0 tensorflow-estimator==2.4.0 termcolor==1.1.0 terminado==0.9.4 testpath==0.4.4 threadpoolctl==2.1.0 tifffile==2020.12.8 tikzplotlib==0.9.8 tkseem==0.0.3 tokenizers==0.10.2 toml==0.10.2 torch==1.13.1 torch-stft==0.1.4 torchaudio==0.13.1 torchmetrics==0.6.0 torchvision==0.14.1 tornado==6.1 tqdm==4.61.1 trackrip==1.2.1 traitlets==5.0.5 transformers==4.15.0 typed-ast==1.4.1 typer==0.4.0 typing-extensions==3.7.4.3 Unidecode==1.3.2 uritemplate==3.0.1 urllib3==1.26.2 virtualenv==20.2.1 wandb==0.12.6 wcwidth==0.2.5 webdataset==0.1.62 webencodings==0.5.1 Werkzeug==1.0.1 wget==3.2 widgetsnbextension==3.5.1 wordninja==2.0.0 wrapt==1.12.1 xmltodict==0.13.0 xxhash==2.0.0 yamllint==1.23.0 yarg==0.1.9 yarl==1.7.2 yaspin==2.1.0 youtokentome==1.0.6 youtube-dl==2021.6.6 zipp==3.6.0 2023-01-07 15:57:02,047 - speechbrain.core - ERROR - Exception: Traceback (most recent call last): File "ctc_train.py", line 305, in train_data, valid_data, test_datasets, label_encoder = dataio_prepare( File "ctc_train.py", line 178, in dataio_prepare train_data = sb.dataio.dataset.DynamicItemDataset.from_csv( File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/dataio/dataset.py", line 365, in from_csv data = load_data_csv(csv_path, replacements) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/dataio/dataio.py", line 127, in load_data_csv with open(csv_path, newline="") as csvfile: FileNotFoundError: [Errno 2] No such file or directory: 'test_salah.csv' 2023-01-07 15:57:54,234 - speechbrain.core - INFO - Beginning experiment! 2023-01-07 15:57:54,234 - speechbrain.core - INFO - Experiment folder: partly_frozen_splitted_wavlm 2023-01-07 15:57:54,891 - speechbrain.utils.superpowers - DEBUG - abkhazia==1.0 absl-py==0.11.0 aiohttp==3.8.0 aiosignal==1.2.0 alabaster==0.7.12 alembic==1.7.4 altgraph==0.17 antlr4-python3-runtime==4.8 appdirs==1.4.4 argcomplete==1.12.2 argon2-cffi==20.1.0 asgiref==3.6.0 astunparse==1.6.3 async-generator==1.10 async-timeout==4.0.0 attrdict==2.0.1 attrs==20.3.0 audeer==1.16.0 audformat==0.11.5 audinterface==0.7.0 audiofile==1.0.0 audiomentations==0.25.0 audioread==2.1.9 audobject==0.4.14 audresample==0.1.6 -e git+https://github.com/facebookresearch/WavAugment.git@54afcdb00ccc852c2f030f239f8532c9562b550e#egg=augment autopage==0.4.0 Babel==2.9.0 backcall==0.2.0 beautifulsoup4==4.10.0 black==19.10b0 bleach==3.3.0 boto3==1.20.2 botocore==1.23.2 braceexpand==0.1.7 cachetools==4.2.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.14.3 cfgv==3.2.0 chardet==3.0.4 charset-normalizer==2.0.7 click==7.1.2 cliff==3.9.0 clldutils==3.5.4 cmaes==0.8.2 cmake==3.18.4.post1 cmd2==2.2.0 colorama==0.4.4 colorlog==4.6.2 configparser==5.1.0 cryptography==38.0.4 csvw==1.8.1 cycler==0.10.0 Cython==0.29.21 dataclasses==0.6 datasets==1.5.0 decorator==4.4.2 deepspeech==0.9.1 defusedxml==0.7.1 denoiser==0.1.5 dill==0.3.3 Distance==0.1.3 distlib==0.3.1 Django==3.2.16 django-auditlog==2.2.1 django-filter==22.1 django-js-asset==1.2.2 django-mptt==0.14.0 djangorestframework==3.14.0 docker-pycreds==0.4.0 docopt==0.6.2 docutils==0.16 drf-excel==2.2.0 drf-flex-fields==1.0.0 drf-renderer-xlsx==0.4.1 easyocr==1.2.1 editdistance==0.6.0 emoji==2.2.0 entrypoints==0.3 et-xmlfile==1.1.0 exceptiongroup==1.1.0 farasapy==0.0.14 fasttext==0.9.2 ffmpeg-python==0.2.0 filelock==3.0.12 flake8==3.7.9 flatbuffers==1.12 frozendict==2.0.7 frozenlist==1.2.0 fsspec==2021.11.0 future==0.18.2 g2p-en==2.1.0 gast==0.3.3 gdown==4.2.0 gensim==4.0.1 gitdb==4.0.9 GitPython==3.1.24 google-auth==1.24.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 greenlet==1.1.2 grpcio==1.32.0 h5features==1.3.2 h5py==2.10.0 htk-io==0.5 huggingface-hub==0.9.1 hydra-colorlog==0.1.4 hydra-core==0.11.3 HyperPyYAML==1.1.0 hypothesis==6.61.2 identify==1.5.10 idna==2.10 imageio==2.9.0 imagesize==1.2.0 importlib-metadata==4.8.1 importlib-resources==5.2.2 inflect==5.3.0 ipadic==1.0.0 ipykernel==5.3.4 ipython==7.19.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 iso-639==0.4.5 isodate==0.6.0 isort==4.3.21 jedi==0.17.2 jieba==0.42.1 Jinja2==2.11.2 jiwer==2.2.0 jmespath==0.10.0 joblib==0.17.0 jsonschema==3.2.0 julius==0.2.7 jupyter-client==6.1.7 jupyter-core==4.7.0 jupyterlab-pygments==0.1.2 jupyterlab-widgets==1.0.0 kaitaistruct==0.9 kaldi-io==0.9.4 kaldi-python-io==1.2.2 kaldiio==2.17.2 kenlm @ https://github.com/kpu/kenlm/archive/master.zip Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 lang-trans==0.6.0 latexcodec==2.0.1 ldap3==2.9.1 librosa==0.9.0 llvmlite==0.35.0 lxml==4.9.0 Mako==1.1.5 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.14.0 matplotlib==3.3.3 mccabe==0.6.1 mcd==0.4 mecab-python3==1.0.3 megatron-lm==2.2.0 mido==1.2.10 mistune==0.8.4 more-itertools==8.6.0 mpmath==1.2.1 multidict==5.2.0 multiprocess==0.70.11.1 nbclient==0.5.3 nbconvert==6.0.7 nbformat==5.1.3 NEMO==4.3.2 nemo-toolkit==1.4.0 nest-asyncio==1.5.1 networkx==2.5 nltk==3.5 nodeenv==1.5.0 notebook==6.3.0 numba==0.52.0 numpy==1.19.4 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 oauthlib==3.1.0 omegaconf==1.4.1 onnx==1.10.2 OpenCC==1.1.2 opencv-python==4.4.0.46 openpyxl==3.0.9 opensmile==2.2.0 opt-einsum==3.3.0 optuna==2.10.0 oyaml==1.0 packaging==22.0 pandas==1.2.5 pandocfilters==1.4.3 pangu==4.0.6.1 parameterized==0.8.1 parso==0.7.1 pathspec==0.8.1 pathtools==0.1.2 pbr==5.6.0 pefile==2019.4.18 pescador==2.1.0 pesq==0.0.3 pexpect==4.8.0 phonemizer==2.2.1 pickleshare==0.7.5 Pillow==9.3.0 pip-api==0.0.23 pipreqs==0.4.11 pluggy==0.13.1 pooch==1.3.0 portalocker==2.3.2 pre-commit==2.9.0 pretty-midi==0.2.9 prettytable==2.2.1 progressbar2==3.53.1 prometheus-client==0.10.1 promise==2.3 prompt-toolkit==3.0.8 protobuf==3.14.0 psutil==5.6.6 ptyprocess==0.6.0 py==1.9.0 py-espeak-ng==0.1.8 pyannote.audio==1.1.1 pyannote.core==4.3 pyannote.database==4.1.1 pyannote.metrics==3.1 pyannote.pipeline==1.5.2 PyArabic==0.6.15 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.8.1 pybtex==0.24.0 pybtex-docutils==1.0.1 pycodestyle==2.5.0 pycparser==2.20 pyctcdecode==0.4.0 pyDeprecate==0.3.1 pydub==0.25.1 pyflakes==2.1.1 Pygments==2.7.2 pygtrie==2.5.0 pymodbus==2.5.3 pyparsing==2.4.7 pyperclip==1.8.2 pypinyin==0.43.0 pyrsistent==0.17.3 pyserial==3.5 PySocks==1.7.1 pystoi==0.3.3 pytest==5.4.1 pytest-runner==5.3.1 python-bidi==0.4.2 python-crfsuite==0.9.7 python-dateutil==2.8.2 python-Levenshtein==0.12.2 python-utils==2.4.0 pytorch-lightning==1.4.9 pytube==11.0.1 pytz==2022.6 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==20.0.0 rapidfuzz==1.8.2 regex==2020.11.13 requests==2.28.1 requests-oauthlib==1.3.0 resampy==0.2.2 rfc3986==1.4.0 rsa==4.7 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 s3m==1.1.0 s3transfer==0.5.0 sacrebleu==2.0.0 sacremoses==0.0.44 scikit-image==0.18.1 scikit-learn==0.23.2 scipy==1.5.4 -e git+https://github.com/sanghack81/SDCIT@00d060dde733fde9345154a494f81e97fb395ca7#egg=SDCIT seaborn==0.11.1 segments==2.1.3 Send2Trash==1.5.0 sentencepiece==0.1.94 sentry-sdk==1.4.3 shellingham==1.4.0 shortuuid==1.0.7 SIDEKIT==1.3.8.5.2 simplejson==3.17.5 six==1.15.0 smart-open==5.0.0 smmap==5.0.0 snowballstemmer==2.0.0 sortedcollections==2.1.0 sortedcontainers==2.4.0 sounddevice==0.4.5 SoundFile==0.10.3.post1 soupsieve==2.3 sox==1.4.1 sparsemax==0.1.9 speechbrain==0.5.13 sphfile==1.0.3 Sphinx==3.3.1 sphinx-rtd-theme==0.4.3 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-bibtex==2.4.1 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 SQLAlchemy==1.4.25 sqlparse==0.4.2 stanza==1.4.2 stevedore==3.4.0 subprocess32==3.5.4 sympy==1.9 tabulate==0.8.9 tensorboard==2.4.0 tensorboard-plugin-wit==1.7.0 tensorflow==2.4.0 tensorflow-estimator==2.4.0 termcolor==1.1.0 terminado==0.9.4 testpath==0.4.4 threadpoolctl==2.1.0 tifffile==2020.12.8 tikzplotlib==0.9.8 tkseem==0.0.3 tokenizers==0.10.2 toml==0.10.2 torch==1.13.1 torch-stft==0.1.4 torchaudio==0.13.1 torchmetrics==0.6.0 torchvision==0.14.1 tornado==6.1 tqdm==4.61.1 trackrip==1.2.1 traitlets==5.0.5 transformers==4.15.0 typed-ast==1.4.1 typer==0.4.0 typing-extensions==3.7.4.3 Unidecode==1.3.2 uritemplate==3.0.1 urllib3==1.26.2 virtualenv==20.2.1 wandb==0.12.6 wcwidth==0.2.5 webdataset==0.1.62 webencodings==0.5.1 Werkzeug==1.0.1 wget==3.2 widgetsnbextension==3.5.1 wordninja==2.0.0 wrapt==1.12.1 xmltodict==0.13.0 xxhash==2.0.0 yamllint==1.23.0 yarg==0.1.9 yarl==1.7.2 yaspin==2.1.0 youtokentome==1.0.6 youtube-dl==2021.6.6 zipp==3.6.0 2023-01-07 15:57:54,987 - speechbrain.dataio.encoder - DEBUG - Would load categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt, but file doesn't exist yet. 2023-01-07 15:57:54,988 - speechbrain.dataio.encoder - INFO - Moving label 'ت' from index 0, because '' was put at its place. 2023-01-07 15:57:54,988 - speechbrain.dataio.encoder - INFO - Moving label 'ع' from index 1, because '' was put at its place. 2023-01-07 15:57:54,989 - speechbrain.dataio.encoder - INFO - Moving label 'ب' from index 2, because '' was put at its place. 2023-01-07 15:57:54,989 - speechbrain.dataio.encoder - INFO - Load called, but CTCTextEncoder is not empty. Loaded data will overwrite everything. This is normal if there is e.g. an unk label defined at init. 2023-01-07 15:57:54,990 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 15:57:54,990 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used 2023-01-07 15:57:54,990 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used 2023-01-07 15:57:57,073 - speechbrain.core - INFO - 313.4M trainable parameters in ASR 2023-01-07 15:57:57,075 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet. 2023-01-07 15:57:57,075 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1 2023-01-07 15:57:57,132 - speechbrain.core - ERROR - Exception: Traceback (most recent call last): File "ctc_train.py", line 322, in asr_brain.fit( File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1153, in fit self._fit_train(train_set=train_set, epoch=epoch, enable=enable) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1009, in _fit_train loss = self.fit_batch(batch) File "ctc_train.py", line 88, in fit_batch predictions = self.compute_forward(batch, sb.Stage.TRAIN) File "ctc_train.py", line 42, in compute_forward feats = self.modules.wav2vec2(wavs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/lobes/models/huggingface_wav2vec.py", line 266, in forward return self.extract_features(wav) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/lobes/models/huggingface_wav2vec.py", line 281, in extract_features out = self.model(wav)[0] File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/transformers/models/wavlm/modeling_wavlm.py", line 1232, in forward extract_features = self.feature_extractor(input_values) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/transformers/models/wavlm/modeling_wavlm.py", line 400, in forward hidden_states = conv_layer(hidden_states) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/transformers/models/wavlm/modeling_wavlm.py", line 270, in forward hidden_states = self.conv(hidden_states) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1194, in _call_impl return forward_call(*input, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 313, in forward return self._conv_forward(input, self.weight, self.bias) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/nn/modules/conv.py", line 309, in _conv_forward return F.conv1d(input, weight, bias, self.stride, RuntimeError: Expected 2D (unbatched) or 3D (batched) input to conv1d, but got input of size: [2, 1, 168960, 1] 2023-01-07 15:59:22,733 - speechbrain.core - INFO - Beginning experiment! 2023-01-07 15:59:22,733 - speechbrain.core - INFO - Experiment folder: partly_frozen_splitted_wavlm 2023-01-07 15:59:23,373 - speechbrain.utils.superpowers - DEBUG - abkhazia==1.0 absl-py==0.11.0 aiohttp==3.8.0 aiosignal==1.2.0 alabaster==0.7.12 alembic==1.7.4 altgraph==0.17 antlr4-python3-runtime==4.8 appdirs==1.4.4 argcomplete==1.12.2 argon2-cffi==20.1.0 asgiref==3.6.0 astunparse==1.6.3 async-generator==1.10 async-timeout==4.0.0 attrdict==2.0.1 attrs==20.3.0 audeer==1.16.0 audformat==0.11.5 audinterface==0.7.0 audiofile==1.0.0 audiomentations==0.25.0 audioread==2.1.9 audobject==0.4.14 audresample==0.1.6 -e git+https://github.com/facebookresearch/WavAugment.git@54afcdb00ccc852c2f030f239f8532c9562b550e#egg=augment autopage==0.4.0 Babel==2.9.0 backcall==0.2.0 beautifulsoup4==4.10.0 black==19.10b0 bleach==3.3.0 boto3==1.20.2 botocore==1.23.2 braceexpand==0.1.7 cachetools==4.2.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.14.3 cfgv==3.2.0 chardet==3.0.4 charset-normalizer==2.0.7 click==7.1.2 cliff==3.9.0 clldutils==3.5.4 cmaes==0.8.2 cmake==3.18.4.post1 cmd2==2.2.0 colorama==0.4.4 colorlog==4.6.2 configparser==5.1.0 cryptography==38.0.4 csvw==1.8.1 cycler==0.10.0 Cython==0.29.21 dataclasses==0.6 datasets==1.5.0 decorator==4.4.2 deepspeech==0.9.1 defusedxml==0.7.1 denoiser==0.1.5 dill==0.3.3 Distance==0.1.3 distlib==0.3.1 Django==3.2.16 django-auditlog==2.2.1 django-filter==22.1 django-js-asset==1.2.2 django-mptt==0.14.0 djangorestframework==3.14.0 docker-pycreds==0.4.0 docopt==0.6.2 docutils==0.16 drf-excel==2.2.0 drf-flex-fields==1.0.0 drf-renderer-xlsx==0.4.1 easyocr==1.2.1 editdistance==0.6.0 emoji==2.2.0 entrypoints==0.3 et-xmlfile==1.1.0 exceptiongroup==1.1.0 farasapy==0.0.14 fasttext==0.9.2 ffmpeg-python==0.2.0 filelock==3.0.12 flake8==3.7.9 flatbuffers==1.12 frozendict==2.0.7 frozenlist==1.2.0 fsspec==2021.11.0 future==0.18.2 g2p-en==2.1.0 gast==0.3.3 gdown==4.2.0 gensim==4.0.1 gitdb==4.0.9 GitPython==3.1.24 google-auth==1.24.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 greenlet==1.1.2 grpcio==1.32.0 h5features==1.3.2 h5py==2.10.0 htk-io==0.5 huggingface-hub==0.9.1 hydra-colorlog==0.1.4 hydra-core==0.11.3 HyperPyYAML==1.1.0 hypothesis==6.61.2 identify==1.5.10 idna==2.10 imageio==2.9.0 imagesize==1.2.0 importlib-metadata==4.8.1 importlib-resources==5.2.2 inflect==5.3.0 ipadic==1.0.0 ipykernel==5.3.4 ipython==7.19.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 iso-639==0.4.5 isodate==0.6.0 isort==4.3.21 jedi==0.17.2 jieba==0.42.1 Jinja2==2.11.2 jiwer==2.2.0 jmespath==0.10.0 joblib==0.17.0 jsonschema==3.2.0 julius==0.2.7 jupyter-client==6.1.7 jupyter-core==4.7.0 jupyterlab-pygments==0.1.2 jupyterlab-widgets==1.0.0 kaitaistruct==0.9 kaldi-io==0.9.4 kaldi-python-io==1.2.2 kaldiio==2.17.2 kenlm @ https://github.com/kpu/kenlm/archive/master.zip Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 lang-trans==0.6.0 latexcodec==2.0.1 ldap3==2.9.1 librosa==0.9.0 llvmlite==0.35.0 lxml==4.9.0 Mako==1.1.5 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.14.0 matplotlib==3.3.3 mccabe==0.6.1 mcd==0.4 mecab-python3==1.0.3 megatron-lm==2.2.0 mido==1.2.10 mistune==0.8.4 more-itertools==8.6.0 mpmath==1.2.1 multidict==5.2.0 multiprocess==0.70.11.1 nbclient==0.5.3 nbconvert==6.0.7 nbformat==5.1.3 NEMO==4.3.2 nemo-toolkit==1.4.0 nest-asyncio==1.5.1 networkx==2.5 nltk==3.5 nodeenv==1.5.0 notebook==6.3.0 numba==0.52.0 numpy==1.19.4 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 oauthlib==3.1.0 omegaconf==1.4.1 onnx==1.10.2 OpenCC==1.1.2 opencv-python==4.4.0.46 openpyxl==3.0.9 opensmile==2.2.0 opt-einsum==3.3.0 optuna==2.10.0 oyaml==1.0 packaging==22.0 pandas==1.2.5 pandocfilters==1.4.3 pangu==4.0.6.1 parameterized==0.8.1 parso==0.7.1 pathspec==0.8.1 pathtools==0.1.2 pbr==5.6.0 pefile==2019.4.18 pescador==2.1.0 pesq==0.0.3 pexpect==4.8.0 phonemizer==2.2.1 pickleshare==0.7.5 Pillow==9.3.0 pip-api==0.0.23 pipreqs==0.4.11 pluggy==0.13.1 pooch==1.3.0 portalocker==2.3.2 pre-commit==2.9.0 pretty-midi==0.2.9 prettytable==2.2.1 progressbar2==3.53.1 prometheus-client==0.10.1 promise==2.3 prompt-toolkit==3.0.8 protobuf==3.14.0 psutil==5.6.6 ptyprocess==0.6.0 py==1.9.0 py-espeak-ng==0.1.8 pyannote.audio==1.1.1 pyannote.core==4.3 pyannote.database==4.1.1 pyannote.metrics==3.1 pyannote.pipeline==1.5.2 PyArabic==0.6.15 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.8.1 pybtex==0.24.0 pybtex-docutils==1.0.1 pycodestyle==2.5.0 pycparser==2.20 pyctcdecode==0.4.0 pyDeprecate==0.3.1 pydub==0.25.1 pyflakes==2.1.1 Pygments==2.7.2 pygtrie==2.5.0 pymodbus==2.5.3 pyparsing==2.4.7 pyperclip==1.8.2 pypinyin==0.43.0 pyrsistent==0.17.3 pyserial==3.5 PySocks==1.7.1 pystoi==0.3.3 pytest==5.4.1 pytest-runner==5.3.1 python-bidi==0.4.2 python-crfsuite==0.9.7 python-dateutil==2.8.2 python-Levenshtein==0.12.2 python-utils==2.4.0 pytorch-lightning==1.4.9 pytube==11.0.1 pytz==2022.6 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==20.0.0 rapidfuzz==1.8.2 regex==2020.11.13 requests==2.28.1 requests-oauthlib==1.3.0 resampy==0.2.2 rfc3986==1.4.0 rsa==4.7 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 s3m==1.1.0 s3transfer==0.5.0 sacrebleu==2.0.0 sacremoses==0.0.44 scikit-image==0.18.1 scikit-learn==0.23.2 scipy==1.5.4 -e git+https://github.com/sanghack81/SDCIT@00d060dde733fde9345154a494f81e97fb395ca7#egg=SDCIT seaborn==0.11.1 segments==2.1.3 Send2Trash==1.5.0 sentencepiece==0.1.94 sentry-sdk==1.4.3 shellingham==1.4.0 shortuuid==1.0.7 SIDEKIT==1.3.8.5.2 simplejson==3.17.5 six==1.15.0 smart-open==5.0.0 smmap==5.0.0 snowballstemmer==2.0.0 sortedcollections==2.1.0 sortedcontainers==2.4.0 sounddevice==0.4.5 SoundFile==0.10.3.post1 soupsieve==2.3 sox==1.4.1 sparsemax==0.1.9 speechbrain==0.5.13 sphfile==1.0.3 Sphinx==3.3.1 sphinx-rtd-theme==0.4.3 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-bibtex==2.4.1 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 SQLAlchemy==1.4.25 sqlparse==0.4.2 stanza==1.4.2 stevedore==3.4.0 subprocess32==3.5.4 sympy==1.9 tabulate==0.8.9 tensorboard==2.4.0 tensorboard-plugin-wit==1.7.0 tensorflow==2.4.0 tensorflow-estimator==2.4.0 termcolor==1.1.0 terminado==0.9.4 testpath==0.4.4 threadpoolctl==2.1.0 tifffile==2020.12.8 tikzplotlib==0.9.8 tkseem==0.0.3 tokenizers==0.10.2 toml==0.10.2 torch==1.13.1 torch-stft==0.1.4 torchaudio==0.13.1 torchmetrics==0.6.0 torchvision==0.14.1 tornado==6.1 tqdm==4.61.1 trackrip==1.2.1 traitlets==5.0.5 transformers==4.15.0 typed-ast==1.4.1 typer==0.4.0 typing-extensions==3.7.4.3 Unidecode==1.3.2 uritemplate==3.0.1 urllib3==1.26.2 virtualenv==20.2.1 wandb==0.12.6 wcwidth==0.2.5 webdataset==0.1.62 webencodings==0.5.1 Werkzeug==1.0.1 wget==3.2 widgetsnbextension==3.5.1 wordninja==2.0.0 wrapt==1.12.1 xmltodict==0.13.0 xxhash==2.0.0 yamllint==1.23.0 yarg==0.1.9 yarl==1.7.2 yaspin==2.1.0 youtokentome==1.0.6 youtube-dl==2021.6.6 zipp==3.6.0 2023-01-07 15:59:23,493 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 15:59:23,493 - speechbrain.dataio.encoder - INFO - Load called, but CTCTextEncoder is not empty. Loaded data will overwrite everything. This is normal if there is e.g. an unk label defined at init. 2023-01-07 15:59:23,494 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 15:59:23,495 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used 2023-01-07 15:59:23,495 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used 2023-01-07 15:59:24,946 - speechbrain.core - INFO - 313.4M trainable parameters in ASR 2023-01-07 15:59:24,949 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet. 2023-01-07 15:59:24,949 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1 2023-01-07 15:59:27,528 - speechbrain.core - ERROR - Exception: Traceback (most recent call last): File "ctc_train.py", line 322, in asr_brain.fit( File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1153, in fit self._fit_train(train_set=train_set, epoch=epoch, enable=enable) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1009, in _fit_train loss = self.fit_batch(batch) File "ctc_train.py", line 92, in fit_batch self.wav2vec_optimizer.step() File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad ret = func(self, *args, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/adam.py", line 220, in step state['exp_avg_sq'] = torch.zeros_like(p, memory_format=torch.preserve_format) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 MiB (GPU 0; 5.80 GiB total capacity; 3.82 GiB already allocated; 52.50 MiB free; 3.94 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-01-07 15:59:54,237 - speechbrain.core - INFO - Beginning experiment! 2023-01-07 15:59:54,237 - speechbrain.core - INFO - Experiment folder: partly_frozen_splitted_wavlm 2023-01-07 15:59:54,868 - speechbrain.utils.superpowers - DEBUG - abkhazia==1.0 absl-py==0.11.0 aiohttp==3.8.0 aiosignal==1.2.0 alabaster==0.7.12 alembic==1.7.4 altgraph==0.17 antlr4-python3-runtime==4.8 appdirs==1.4.4 argcomplete==1.12.2 argon2-cffi==20.1.0 asgiref==3.6.0 astunparse==1.6.3 async-generator==1.10 async-timeout==4.0.0 attrdict==2.0.1 attrs==20.3.0 audeer==1.16.0 audformat==0.11.5 audinterface==0.7.0 audiofile==1.0.0 audiomentations==0.25.0 audioread==2.1.9 audobject==0.4.14 audresample==0.1.6 -e git+https://github.com/facebookresearch/WavAugment.git@54afcdb00ccc852c2f030f239f8532c9562b550e#egg=augment autopage==0.4.0 Babel==2.9.0 backcall==0.2.0 beautifulsoup4==4.10.0 black==19.10b0 bleach==3.3.0 boto3==1.20.2 botocore==1.23.2 braceexpand==0.1.7 cachetools==4.2.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.14.3 cfgv==3.2.0 chardet==3.0.4 charset-normalizer==2.0.7 click==7.1.2 cliff==3.9.0 clldutils==3.5.4 cmaes==0.8.2 cmake==3.18.4.post1 cmd2==2.2.0 colorama==0.4.4 colorlog==4.6.2 configparser==5.1.0 cryptography==38.0.4 csvw==1.8.1 cycler==0.10.0 Cython==0.29.21 dataclasses==0.6 datasets==1.5.0 decorator==4.4.2 deepspeech==0.9.1 defusedxml==0.7.1 denoiser==0.1.5 dill==0.3.3 Distance==0.1.3 distlib==0.3.1 Django==3.2.16 django-auditlog==2.2.1 django-filter==22.1 django-js-asset==1.2.2 django-mptt==0.14.0 djangorestframework==3.14.0 docker-pycreds==0.4.0 docopt==0.6.2 docutils==0.16 drf-excel==2.2.0 drf-flex-fields==1.0.0 drf-renderer-xlsx==0.4.1 easyocr==1.2.1 editdistance==0.6.0 emoji==2.2.0 entrypoints==0.3 et-xmlfile==1.1.0 exceptiongroup==1.1.0 farasapy==0.0.14 fasttext==0.9.2 ffmpeg-python==0.2.0 filelock==3.0.12 flake8==3.7.9 flatbuffers==1.12 frozendict==2.0.7 frozenlist==1.2.0 fsspec==2021.11.0 future==0.18.2 g2p-en==2.1.0 gast==0.3.3 gdown==4.2.0 gensim==4.0.1 gitdb==4.0.9 GitPython==3.1.24 google-auth==1.24.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 greenlet==1.1.2 grpcio==1.32.0 h5features==1.3.2 h5py==2.10.0 htk-io==0.5 huggingface-hub==0.9.1 hydra-colorlog==0.1.4 hydra-core==0.11.3 HyperPyYAML==1.1.0 hypothesis==6.61.2 identify==1.5.10 idna==2.10 imageio==2.9.0 imagesize==1.2.0 importlib-metadata==4.8.1 importlib-resources==5.2.2 inflect==5.3.0 ipadic==1.0.0 ipykernel==5.3.4 ipython==7.19.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 iso-639==0.4.5 isodate==0.6.0 isort==4.3.21 jedi==0.17.2 jieba==0.42.1 Jinja2==2.11.2 jiwer==2.2.0 jmespath==0.10.0 joblib==0.17.0 jsonschema==3.2.0 julius==0.2.7 jupyter-client==6.1.7 jupyter-core==4.7.0 jupyterlab-pygments==0.1.2 jupyterlab-widgets==1.0.0 kaitaistruct==0.9 kaldi-io==0.9.4 kaldi-python-io==1.2.2 kaldiio==2.17.2 kenlm @ https://github.com/kpu/kenlm/archive/master.zip Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 lang-trans==0.6.0 latexcodec==2.0.1 ldap3==2.9.1 librosa==0.9.0 llvmlite==0.35.0 lxml==4.9.0 Mako==1.1.5 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.14.0 matplotlib==3.3.3 mccabe==0.6.1 mcd==0.4 mecab-python3==1.0.3 megatron-lm==2.2.0 mido==1.2.10 mistune==0.8.4 more-itertools==8.6.0 mpmath==1.2.1 multidict==5.2.0 multiprocess==0.70.11.1 nbclient==0.5.3 nbconvert==6.0.7 nbformat==5.1.3 NEMO==4.3.2 nemo-toolkit==1.4.0 nest-asyncio==1.5.1 networkx==2.5 nltk==3.5 nodeenv==1.5.0 notebook==6.3.0 numba==0.52.0 numpy==1.19.4 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 oauthlib==3.1.0 omegaconf==1.4.1 onnx==1.10.2 OpenCC==1.1.2 opencv-python==4.4.0.46 openpyxl==3.0.9 opensmile==2.2.0 opt-einsum==3.3.0 optuna==2.10.0 oyaml==1.0 packaging==22.0 pandas==1.2.5 pandocfilters==1.4.3 pangu==4.0.6.1 parameterized==0.8.1 parso==0.7.1 pathspec==0.8.1 pathtools==0.1.2 pbr==5.6.0 pefile==2019.4.18 pescador==2.1.0 pesq==0.0.3 pexpect==4.8.0 phonemizer==2.2.1 pickleshare==0.7.5 Pillow==9.3.0 pip-api==0.0.23 pipreqs==0.4.11 pluggy==0.13.1 pooch==1.3.0 portalocker==2.3.2 pre-commit==2.9.0 pretty-midi==0.2.9 prettytable==2.2.1 progressbar2==3.53.1 prometheus-client==0.10.1 promise==2.3 prompt-toolkit==3.0.8 protobuf==3.14.0 psutil==5.6.6 ptyprocess==0.6.0 py==1.9.0 py-espeak-ng==0.1.8 pyannote.audio==1.1.1 pyannote.core==4.3 pyannote.database==4.1.1 pyannote.metrics==3.1 pyannote.pipeline==1.5.2 PyArabic==0.6.15 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.8.1 pybtex==0.24.0 pybtex-docutils==1.0.1 pycodestyle==2.5.0 pycparser==2.20 pyctcdecode==0.4.0 pyDeprecate==0.3.1 pydub==0.25.1 pyflakes==2.1.1 Pygments==2.7.2 pygtrie==2.5.0 pymodbus==2.5.3 pyparsing==2.4.7 pyperclip==1.8.2 pypinyin==0.43.0 pyrsistent==0.17.3 pyserial==3.5 PySocks==1.7.1 pystoi==0.3.3 pytest==5.4.1 pytest-runner==5.3.1 python-bidi==0.4.2 python-crfsuite==0.9.7 python-dateutil==2.8.2 python-Levenshtein==0.12.2 python-utils==2.4.0 pytorch-lightning==1.4.9 pytube==11.0.1 pytz==2022.6 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==20.0.0 rapidfuzz==1.8.2 regex==2020.11.13 requests==2.28.1 requests-oauthlib==1.3.0 resampy==0.2.2 rfc3986==1.4.0 rsa==4.7 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 s3m==1.1.0 s3transfer==0.5.0 sacrebleu==2.0.0 sacremoses==0.0.44 scikit-image==0.18.1 scikit-learn==0.23.2 scipy==1.5.4 -e git+https://github.com/sanghack81/SDCIT@00d060dde733fde9345154a494f81e97fb395ca7#egg=SDCIT seaborn==0.11.1 segments==2.1.3 Send2Trash==1.5.0 sentencepiece==0.1.94 sentry-sdk==1.4.3 shellingham==1.4.0 shortuuid==1.0.7 SIDEKIT==1.3.8.5.2 simplejson==3.17.5 six==1.15.0 smart-open==5.0.0 smmap==5.0.0 snowballstemmer==2.0.0 sortedcollections==2.1.0 sortedcontainers==2.4.0 sounddevice==0.4.5 SoundFile==0.10.3.post1 soupsieve==2.3 sox==1.4.1 sparsemax==0.1.9 speechbrain==0.5.13 sphfile==1.0.3 Sphinx==3.3.1 sphinx-rtd-theme==0.4.3 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-bibtex==2.4.1 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 SQLAlchemy==1.4.25 sqlparse==0.4.2 stanza==1.4.2 stevedore==3.4.0 subprocess32==3.5.4 sympy==1.9 tabulate==0.8.9 tensorboard==2.4.0 tensorboard-plugin-wit==1.7.0 tensorflow==2.4.0 tensorflow-estimator==2.4.0 termcolor==1.1.0 terminado==0.9.4 testpath==0.4.4 threadpoolctl==2.1.0 tifffile==2020.12.8 tikzplotlib==0.9.8 tkseem==0.0.3 tokenizers==0.10.2 toml==0.10.2 torch==1.13.1 torch-stft==0.1.4 torchaudio==0.13.1 torchmetrics==0.6.0 torchvision==0.14.1 tornado==6.1 tqdm==4.61.1 trackrip==1.2.1 traitlets==5.0.5 transformers==4.15.0 typed-ast==1.4.1 typer==0.4.0 typing-extensions==3.7.4.3 Unidecode==1.3.2 uritemplate==3.0.1 urllib3==1.26.2 virtualenv==20.2.1 wandb==0.12.6 wcwidth==0.2.5 webdataset==0.1.62 webencodings==0.5.1 Werkzeug==1.0.1 wget==3.2 widgetsnbextension==3.5.1 wordninja==2.0.0 wrapt==1.12.1 xmltodict==0.13.0 xxhash==2.0.0 yamllint==1.23.0 yarg==0.1.9 yarl==1.7.2 yaspin==2.1.0 youtokentome==1.0.6 youtube-dl==2021.6.6 zipp==3.6.0 2023-01-07 15:59:54,960 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 15:59:54,960 - speechbrain.dataio.encoder - INFO - Load called, but CTCTextEncoder is not empty. Loaded data will overwrite everything. This is normal if there is e.g. an unk label defined at init. 2023-01-07 15:59:54,961 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 15:59:54,961 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used 2023-01-07 15:59:54,961 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used 2023-01-07 15:59:56,396 - speechbrain.core - INFO - 313.4M trainable parameters in ASR 2023-01-07 15:59:56,398 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet. 2023-01-07 15:59:56,398 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1 2023-01-07 15:59:57,755 - speechbrain.core - ERROR - Exception: Traceback (most recent call last): File "ctc_train.py", line 322, in asr_brain.fit( File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1153, in fit self._fit_train(train_set=train_set, epoch=epoch, enable=enable) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1009, in _fit_train loss = self.fit_batch(batch) File "ctc_train.py", line 92, in fit_batch self.wav2vec_optimizer.step() File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad ret = func(self, *args, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/adam.py", line 218, in step state['exp_avg'] = torch.zeros_like(p, memory_format=torch.preserve_format) torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 5.80 GiB total capacity; 3.86 GiB already allocated; 64.50 MiB free; 3.93 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF 2023-01-07 16:00:47,395 - speechbrain.core - INFO - Beginning experiment! 2023-01-07 16:00:47,395 - speechbrain.core - INFO - Experiment folder: partly_frozen_splitted_wavlm 2023-01-07 16:00:48,055 - speechbrain.utils.superpowers - DEBUG - abkhazia==1.0 absl-py==0.11.0 aiohttp==3.8.0 aiosignal==1.2.0 alabaster==0.7.12 alembic==1.7.4 altgraph==0.17 antlr4-python3-runtime==4.8 appdirs==1.4.4 argcomplete==1.12.2 argon2-cffi==20.1.0 asgiref==3.6.0 astunparse==1.6.3 async-generator==1.10 async-timeout==4.0.0 attrdict==2.0.1 attrs==20.3.0 audeer==1.16.0 audformat==0.11.5 audinterface==0.7.0 audiofile==1.0.0 audiomentations==0.25.0 audioread==2.1.9 audobject==0.4.14 audresample==0.1.6 -e git+https://github.com/facebookresearch/WavAugment.git@54afcdb00ccc852c2f030f239f8532c9562b550e#egg=augment autopage==0.4.0 Babel==2.9.0 backcall==0.2.0 beautifulsoup4==4.10.0 black==19.10b0 bleach==3.3.0 boto3==1.20.2 botocore==1.23.2 braceexpand==0.1.7 cachetools==4.2.0 certifi @ file:///croot/certifi_1671487769961/work/certifi cffi==1.14.3 cfgv==3.2.0 chardet==3.0.4 charset-normalizer==2.0.7 click==7.1.2 cliff==3.9.0 clldutils==3.5.4 cmaes==0.8.2 cmake==3.18.4.post1 cmd2==2.2.0 colorama==0.4.4 colorlog==4.6.2 configparser==5.1.0 cryptography==38.0.4 csvw==1.8.1 cycler==0.10.0 Cython==0.29.21 dataclasses==0.6 datasets==1.5.0 decorator==4.4.2 deepspeech==0.9.1 defusedxml==0.7.1 denoiser==0.1.5 dill==0.3.3 Distance==0.1.3 distlib==0.3.1 Django==3.2.16 django-auditlog==2.2.1 django-filter==22.1 django-js-asset==1.2.2 django-mptt==0.14.0 djangorestframework==3.14.0 docker-pycreds==0.4.0 docopt==0.6.2 docutils==0.16 drf-excel==2.2.0 drf-flex-fields==1.0.0 drf-renderer-xlsx==0.4.1 easyocr==1.2.1 editdistance==0.6.0 emoji==2.2.0 entrypoints==0.3 et-xmlfile==1.1.0 exceptiongroup==1.1.0 farasapy==0.0.14 fasttext==0.9.2 ffmpeg-python==0.2.0 filelock==3.0.12 flake8==3.7.9 flatbuffers==1.12 frozendict==2.0.7 frozenlist==1.2.0 fsspec==2021.11.0 future==0.18.2 g2p-en==2.1.0 gast==0.3.3 gdown==4.2.0 gensim==4.0.1 gitdb==4.0.9 GitPython==3.1.24 google-auth==1.24.0 google-auth-oauthlib==0.4.2 google-pasta==0.2.0 greenlet==1.1.2 grpcio==1.32.0 h5features==1.3.2 h5py==2.10.0 htk-io==0.5 huggingface-hub==0.9.1 hydra-colorlog==0.1.4 hydra-core==0.11.3 HyperPyYAML==1.1.0 hypothesis==6.61.2 identify==1.5.10 idna==2.10 imageio==2.9.0 imagesize==1.2.0 importlib-metadata==4.8.1 importlib-resources==5.2.2 inflect==5.3.0 ipadic==1.0.0 ipykernel==5.3.4 ipython==7.19.0 ipython-genutils==0.2.0 ipywidgets==7.6.3 iso-639==0.4.5 isodate==0.6.0 isort==4.3.21 jedi==0.17.2 jieba==0.42.1 Jinja2==2.11.2 jiwer==2.2.0 jmespath==0.10.0 joblib==0.17.0 jsonschema==3.2.0 julius==0.2.7 jupyter-client==6.1.7 jupyter-core==4.7.0 jupyterlab-pygments==0.1.2 jupyterlab-widgets==1.0.0 kaitaistruct==0.9 kaldi-io==0.9.4 kaldi-python-io==1.2.2 kaldiio==2.17.2 kenlm @ https://github.com/kpu/kenlm/archive/master.zip Keras-Preprocessing==1.1.2 kiwisolver==1.3.1 lang-trans==0.6.0 latexcodec==2.0.1 ldap3==2.9.1 librosa==0.9.0 llvmlite==0.35.0 lxml==4.9.0 Mako==1.1.5 Markdown==3.3.3 MarkupSafe==1.1.1 marshmallow==3.14.0 matplotlib==3.3.3 mccabe==0.6.1 mcd==0.4 mecab-python3==1.0.3 megatron-lm==2.2.0 mido==1.2.10 mistune==0.8.4 more-itertools==8.6.0 mpmath==1.2.1 multidict==5.2.0 multiprocess==0.70.11.1 nbclient==0.5.3 nbconvert==6.0.7 nbformat==5.1.3 NEMO==4.3.2 nemo-toolkit==1.4.0 nest-asyncio==1.5.1 networkx==2.5 nltk==3.5 nodeenv==1.5.0 notebook==6.3.0 numba==0.52.0 numpy==1.19.4 nvidia-cublas-cu11==11.10.3.66 nvidia-cuda-nvrtc-cu11==11.7.99 nvidia-cuda-runtime-cu11==11.7.99 nvidia-cudnn-cu11==8.5.0.96 oauthlib==3.1.0 omegaconf==1.4.1 onnx==1.10.2 OpenCC==1.1.2 opencv-python==4.4.0.46 openpyxl==3.0.9 opensmile==2.2.0 opt-einsum==3.3.0 optuna==2.10.0 oyaml==1.0 packaging==22.0 pandas==1.2.5 pandocfilters==1.4.3 pangu==4.0.6.1 parameterized==0.8.1 parso==0.7.1 pathspec==0.8.1 pathtools==0.1.2 pbr==5.6.0 pefile==2019.4.18 pescador==2.1.0 pesq==0.0.3 pexpect==4.8.0 phonemizer==2.2.1 pickleshare==0.7.5 Pillow==9.3.0 pip-api==0.0.23 pipreqs==0.4.11 pluggy==0.13.1 pooch==1.3.0 portalocker==2.3.2 pre-commit==2.9.0 pretty-midi==0.2.9 prettytable==2.2.1 progressbar2==3.53.1 prometheus-client==0.10.1 promise==2.3 prompt-toolkit==3.0.8 protobuf==3.14.0 psutil==5.6.6 ptyprocess==0.6.0 py==1.9.0 py-espeak-ng==0.1.8 pyannote.audio==1.1.1 pyannote.core==4.3 pyannote.database==4.1.1 pyannote.metrics==3.1 pyannote.pipeline==1.5.2 PyArabic==0.6.15 pyarrow==3.0.0 pyasn1==0.4.8 pyasn1-modules==0.2.8 pybind11==2.8.1 pybtex==0.24.0 pybtex-docutils==1.0.1 pycodestyle==2.5.0 pycparser==2.20 pyctcdecode==0.4.0 pyDeprecate==0.3.1 pydub==0.25.1 pyflakes==2.1.1 Pygments==2.7.2 pygtrie==2.5.0 pymodbus==2.5.3 pyparsing==2.4.7 pyperclip==1.8.2 pypinyin==0.43.0 pyrsistent==0.17.3 pyserial==3.5 PySocks==1.7.1 pystoi==0.3.3 pytest==5.4.1 pytest-runner==5.3.1 python-bidi==0.4.2 python-crfsuite==0.9.7 python-dateutil==2.8.2 python-Levenshtein==0.12.2 python-utils==2.4.0 pytorch-lightning==1.4.9 pytube==11.0.1 pytz==2022.6 PyWavelets==1.1.1 PyYAML==5.3.1 pyzmq==20.0.0 rapidfuzz==1.8.2 regex==2020.11.13 requests==2.28.1 requests-oauthlib==1.3.0 resampy==0.2.2 rfc3986==1.4.0 rsa==4.7 ruamel.yaml==0.17.21 ruamel.yaml.clib==0.2.7 s3m==1.1.0 s3transfer==0.5.0 sacrebleu==2.0.0 sacremoses==0.0.44 scikit-image==0.18.1 scikit-learn==0.23.2 scipy==1.5.4 -e git+https://github.com/sanghack81/SDCIT@00d060dde733fde9345154a494f81e97fb395ca7#egg=SDCIT seaborn==0.11.1 segments==2.1.3 Send2Trash==1.5.0 sentencepiece==0.1.94 sentry-sdk==1.4.3 shellingham==1.4.0 shortuuid==1.0.7 SIDEKIT==1.3.8.5.2 simplejson==3.17.5 six==1.15.0 smart-open==5.0.0 smmap==5.0.0 snowballstemmer==2.0.0 sortedcollections==2.1.0 sortedcontainers==2.4.0 sounddevice==0.4.5 SoundFile==0.10.3.post1 soupsieve==2.3 sox==1.4.1 sparsemax==0.1.9 speechbrain==0.5.13 sphfile==1.0.3 Sphinx==3.3.1 sphinx-rtd-theme==0.4.3 sphinxcontrib-applehelp==1.0.2 sphinxcontrib-bibtex==2.4.1 sphinxcontrib-devhelp==1.0.2 sphinxcontrib-htmlhelp==1.0.3 sphinxcontrib-jsmath==1.0.1 sphinxcontrib-qthelp==1.0.3 sphinxcontrib-serializinghtml==1.1.4 SQLAlchemy==1.4.25 sqlparse==0.4.2 stanza==1.4.2 stevedore==3.4.0 subprocess32==3.5.4 sympy==1.9 tabulate==0.8.9 tensorboard==2.4.0 tensorboard-plugin-wit==1.7.0 tensorflow==2.4.0 tensorflow-estimator==2.4.0 termcolor==1.1.0 terminado==0.9.4 testpath==0.4.4 threadpoolctl==2.1.0 tifffile==2020.12.8 tikzplotlib==0.9.8 tkseem==0.0.3 tokenizers==0.10.2 toml==0.10.2 torch==1.13.1 torch-stft==0.1.4 torchaudio==0.13.1 torchmetrics==0.6.0 torchvision==0.14.1 tornado==6.1 tqdm==4.61.1 trackrip==1.2.1 traitlets==5.0.5 transformers==4.15.0 typed-ast==1.4.1 typer==0.4.0 typing-extensions==3.7.4.3 Unidecode==1.3.2 uritemplate==3.0.1 urllib3==1.26.2 virtualenv==20.2.1 wandb==0.12.6 wcwidth==0.2.5 webdataset==0.1.62 webencodings==0.5.1 Werkzeug==1.0.1 wget==3.2 widgetsnbextension==3.5.1 wordninja==2.0.0 wrapt==1.12.1 xmltodict==0.13.0 xxhash==2.0.0 yamllint==1.23.0 yarg==0.1.9 yarl==1.7.2 yaspin==2.1.0 youtokentome==1.0.6 youtube-dl==2021.6.6 zipp==3.6.0 2023-01-07 16:00:48,174 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 16:00:48,174 - speechbrain.dataio.encoder - INFO - Load called, but CTCTextEncoder is not empty. Loaded data will overwrite everything. This is normal if there is e.g. an unk label defined at init. 2023-01-07 16:00:48,175 - speechbrain.dataio.encoder - DEBUG - Loaded categorical encoding from partly_frozen_splitted_wavlm/save/label_encoder.txt 2023-01-07 16:00:48,176 - speechbrain.core - INFO - Info: auto_mix_prec arg from hparam file is used 2023-01-07 16:00:48,176 - speechbrain.core - INFO - Info: ckpt_interval_minutes arg from hparam file is used 2023-01-07 16:00:49,674 - speechbrain.core - INFO - 313.4M trainable parameters in ASR 2023-01-07 16:00:50,298 - speechbrain.utils.checkpoints - INFO - Would load a checkpoint here, but none found yet. 2023-01-07 16:00:50,298 - speechbrain.utils.epoch_loop - INFO - Going into epoch 1 2023-01-07 16:01:08,641 - speechbrain.core - ERROR - Exception: Traceback (most recent call last): File "ctc_train.py", line 324, in asr_brain.fit( File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1153, in fit self._fit_train(train_set=train_set, epoch=epoch, enable=enable) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/speechbrain/core.py", line 1009, in _fit_train loss = self.fit_batch(batch) File "ctc_train.py", line 92, in fit_batch self.wav2vec_optimizer.step() File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/optimizer.py", line 23, in _use_grad ret = func(self, *args, **kwargs) File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/adam.py", line 234, in step adam(params_with_grad, File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/adam.py", line 300, in adam func(params, File "/home/salah/anaconda3/envs/salah/lib/python3.8/site-packages/torch/optim/adam.py", line 410, in _single_tensor_adam denom = (exp_avg_sq.sqrt() / bias_correction2_sqrt).add_(eps) KeyboardInterrupt