Correct maximum positional embeddings

The model appears to have been trained with context window = 512, not 2048 as claimed here. This can be seen by looking at the average loss by sequence position on the GPT4 tiny stories dataset (packed into inputs of length 2048):


It would be great to get this changed (for all tinystories models), as the current config is misleading.

config.json

@@ -28,7 +28,7 @@
   "initializer_range": 0.02,
   "intermediate_size": null,
   "layer_norm_epsilon": 1e-05,
-  "max_position_embeddings": 2048,
+  "max_position_embeddings": 512,
   "model_type": "gpt_neo",
   "num_heads": 16,
   "num_layers": 4,