Error when deploying this model via Inference Endpoints
#5
by
edgelesssys
- opened
When deploying this model via inference endpoints in HuggingFace, the following error occurs:
Exit code: 1. Reason: rn loop.run_until_complete(main)
File "/opt/conda/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
return future.result()
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/server.py", line 229, in serve_inner
model = get_model_with_lora_adapters(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 1152, in get_model_with_lora_adapters
model = get_model(
File "/opt/conda/lib/python3.10/site-packages/text_generation_server/models/__init__.py", line 487, in get_model
if max_input_tokens is not None and max_input_tokens <= sliding_window:
TypeError: '<=' not supported between instances of 'int' and 'NoneType'
[2m[3mrank[0m[2m=[0m3[0m
[2m2024-08-07T15:38:14.094704Z[0m [31mERROR[0m [2mtext_generation_launcher[0m[2m:[0m Shard 1 failed to start
[2m2024-08-07T15:38:14.094724Z[0m [32m INFO[0m [2mtext_generation_launcher[0m[2m:[0m Shutting down shards
[2m2024-08-07T15:38:14.096975Z[0m [32m INFO[0m [1mshard-manager[0m: [2mtext_generation_launcher[0m[2m:[0m Terminating shard [2m[3mrank[0m[2m=[0m0[0m
[2m2024-08-07T15:38:14.097001Z[0m [32m INFO[0m [1mshard-manager[0m: [2mtext_generation_launcher[0m[2m:[0m Waiting for shard to gracefully shutdown [2m[3mrank[0m[2m=[0m0[0m
[2m2024-08-07T15:38:14.097313Z[0m [32m INFO[0m [1mshard-manager[0m: [2mtext_generation_launcher[0m[2m:[0m Terminating shard [2m[3mrank[0m[2m=[0m2[0m
[2m2024-08-07T15:38:14.097335Z[0m [32m INFO[0m [1mshard-manager[0m: [2mtext_generation_launcher[0m[2m:[0m Waiting for shard to gracefully shutdown [2m[3mrank[0m[2m=[0m2[0m
[2m2024-08-07T15:38:14.197291Z[0m [32m INFO[0m [1mshard-manager[0m: [2mtext_generation_launcher[0m[2m:[0m shard terminated [2m[3mrank[0m[2m=[0m0[0m
[2m2024-08-07T15:38:14.197578Z[0m [32m INFO[0m [1mshard-manager[0m: [2mtext_generation_launcher[0m[2m:[0m shard terminated [2m[3mrank[0m[2m=[0m2[0m
Error: ShardCannotStart