A small experimental GPT-style language model using routing-based sparse key/value attention to reduce the computational and memory cost of attention at longer context lengths.