# Grok AI source code and weights

Source: https://github.com/xai-org/grok
Kiwi News: https://news.kiwistand.com/stories/Grok-AI-source-code-and-weights?index=0x65f74a8d0ad3fd977e3bee11bd208c57f1e37ab486992069728943bbd3ff24b3ef0ad2ee
Submitted by tim on 2024-03-17 UTC, 6 upvotes, 2 comments

## Summary

This repository provides JAX example code for loading and running Grok-1, an open-weights model, with its weights available via torrent or the HuggingFace Hub. The model has 314B parameters and uses a Mixture of 8 Experts architecture with 2 experts used per token, 64 layers, and a maximum context of 8,192 tokens. The text notes that the MoE implementation is not efficient and was chosen to avoid custom kernels when validating model correctness, and that testing requires a machine with substantial GPU memory. The code and weights are licensed under Apache 2.0.

## Article

This repository contains JAX example code for loading and running the Grok-1 open-weights model.

Make sure to download the checkpoint and place the ckpt-0 directory in checkpoints - see Downloading the weights

Then, run

pip install -r requirements.txt python run.py

to test the code.

The script loads the checkpoint and samples from the model on a test input.

Due to the large size of the model (314B parameters), a machine with enough GPU memory is required to test the model with the example code. The implementation of the MoE layer in this repository is not efficient. The implementation was chosen to avoid the need for custom kernels to validate the correctness of the model.

Model Specifications

Grok-1 is currently designed with the following specifications:

Parameters: 314B

Architecture: Mixture of 8 Experts (MoE)

Experts Utilization: 2 experts used per token

Layers: 64

Attention Heads: 48 for queries, 8 for keys/values

Embedding Size: 6,144

Tokenization: SentencePiece tokenizer with 131,072 tokens

Additional Features:

Rotary embeddings (RoPE)

Supports activation sharding and 8-bit quantization

Maximum Sequence Length (context): 8,192 tokens

Downloading the weights

You can download the weights using a torrent client and this magnet link:

magnet:?xt=urn:btih:5f96d43576e3d386c9ba65b883210a393b68210e&tr=https%3A%2F%2Facademictorrents.com%2Fannounce.php&tr=udp%3A%2F%2Ftracker.coppersurfer.tk%3A6969&tr=udp%3A%2F%2Ftracker.opentrackr.org%3A1337%2Fannounce

or directly using HuggingFace 🤗 Hub:

git clone https://github.com/xai-org/grok-1.git && cd grok-1 pip install huggingface_hub[hf_transfer] huggingface-cli download xai-org/grok-1 --repo-type model --include ckpt-0/* --local-dir checkpoints --local-dir-use-symlinks False

License

The code and associated Grok-1 weights in this release are licensed under the Apache 2.0 license. The license only applies to the source files in this repository and the model weights of Grok-1.

## Comments

**tim**: I know other people have said this too, but it is true: that model's source code is remarkably lean considering the complexity the tool can handle. This is really cool!

**m-j-r.eth**: I like that there's pressure to be any degree more transparent. Grok is a sparse model, only 8-bit precision weights, but even this is more information than what GPT-4 has presented. I'd love to see the performance after tacking on the Hermes function-calling ( https://github.com/NousResearch/Hermes-Function-Calling ).

gotta stock up on popcorn, though, because who knows what releases in the coming weeks.