Codex changed how developers write software by turning natural language into working code. This article breaks down how it works, what it can do, where it falls short, and how today's more capable code models have picked up where it left off.
OpenAI's Codex arrived in 2021 and instantly reframed what developers thought was possible. Before it, autocomplete meant finishing a variable name. After it, you could type a plain English sentence and watch a working function appear. That shift from syntax assistance to intent translation is what this article unpacks.
Codex is a large language model trained on both natural language text and an enormous corpus of source code scraped from public repositories, most notably GitHub. It is, at its core, a descendant of GPT-3, fine-tuned specifically to predict code tokens rather than prose.
The training data gave it something remarkable: statistical knowledge of how humans write software. Not just syntax rules, but common idioms, library conventions, how functions are typically named, and how comments relate to the code below them. It learned programming not from formal specifications but from millions of examples of programmers actually doing their work.
The architecture is a transformer, the same family of neural networks behind every major language model today. Transformers use a mechanism called self-attention to weigh the relevance of every token in the input context against every other token. For code, this is powerful: a variable defined 200 lines earlier is still "visible" to the model when it predicts what comes next.
Codex shipped in two primary configurations: a smaller cushman variant (roughly 12B parameters) optimized for speed, and a larger davinci variant focused on accuracy. The davinci variant powered the original GitHub Copilot and was the configuration that genuinely impressed people when it launched.
General-purpose LLMs are trained to handle everything: essays, questions, summaries, conversations. Codex traded breadth for depth. Its fine-tuning on code means it reliably produces syntactically valid output in dozens of programming languages, handles code-specific context better than prose-focused models, and understands abstract programming concepts like recursion, async execution, and API contracts at a practical level.
💡 A general LLM will describe a sorting algorithm. Codex will write one, with edge cases handled.
At a mechanical level, code generation is next-token prediction. You give the model a prompt, which might be a comment, a function signature, or an existing file, and the model calculates a probability distribution over every possible next token, then samples from that distribution.
What makes this powerful for code is that valid programs occupy only a tiny fraction of all possible token sequences. Training on real code teaches the model the shape of that valid space, so its predictions land inside it most of the time.
Code models operate within a context window, a maximum number of tokens they can process at once. For Codex, this was 4,096 or 8,192 tokens depending on the variant. That limits how much surrounding code the model can "see" when making a prediction.
Modern code models have pushed this dramatically further. GPT 4.1 handles up to 1 million token contexts, meaning entire codebases can fit in a single prompt. Granite 8B Code Instruct 128K from IBM offers 128,000 tokens specifically optimized for code tasks, a dramatic improvement over Codex's original limits.
The apparent "magic" of Codex is that it was trained on code repositories containing both comments and implementations side by side. It learned that a comment like # parse a JSON file and return a dict typically precedes a specific type of Python function. Feeding it that comment activates the association and it predicts the corresponding implementation.
This is not reasoning in any philosophical sense. It is extraordinarily powerful pattern matching operating at a scale that produces outputs indistinguishable from understanding.
💡 Worth noting: Codex is not executing code or checking it against a runtime. Every suggestion is a statistical prediction. This is why AI-generated code still needs human review.
Understanding the actual capability profile stops you from expecting too much or too little from any AI code model.
Codex was sunset by OpenAI in March 2023. The code generation landscape has moved dramatically since then, and every meaningful improvement traces back to two axes: larger context windows and integrated reasoning.
| Model | Context Window | Specialization | Available |
|---|---|---|---|
| Codex (davinci) | 8K tokens | Code focused | Deprecated |
| GPT 4.1 | 1M tokens | General + Code | PicassoIA |
| Granite 8B Code 128K | 128K tokens | Code only | PicassoIA |
| Granite 20B Code 8K | 8K tokens | Code only | PicassoIA |
| DeepSeek R1 | 128K tokens | Code + Reasoning | PicassoIA |
| Claude 4 Sonnet | 200K tokens | Code + Writing | PicassoIA |
| Kimi K2 Instruct | 128K tokens | Agentic Coding | PicassoIA |
Codex was optimized for a single task: predict what code comes next. Newer models combine code generation with chain-of-thought reasoning, working through a problem step by step before producing code. This closes the gap on multi-step programming tasks where pure pattern matching is insufficient.
DeepSeek R1 stands out here: it produces visible reasoning traces, so you can follow exactly why it structured code a particular way. Claude 4 Sonnet similarly explains its own code in natural language without being prompted, which makes reviewing AI-generated output substantially faster.
The models that surpassed Codex are all available through PicassoIA's large language model collection. No API keys to manage, no local environment setup. Here is how to put them to work on real code generation tasks.
Granite 8B Code Instruct 128K is IBM's dedicated code model, trained specifically on programming tasks including generation, debugging, explanation, and refactoring.
💡 Tip: For refactoring tasks, paste the existing function first, then add: "Rewrite this to handle [edge case] while keeping the same interface."
GPT 4.1 excels when the task spans multiple files. Its 1M token window means you can paste entire modules or multiple related files together and ask it to reason across all of them at once. This is the task where Codex most obviously fell short, and where GPT 4.1 delivers a genuinely different experience.
Use it for:
Developers who get the most out of AI code generation treat it as a fast first draft, not a finished product. Here is what that looks like in practice.
Test generation is where code LLMs add the most practical day-to-day value. Most developers write tests after writing the implementation, and it is tedious. You already know what the function does, you just need to enumerate the cases.
A productive prompt pattern:
Given this function:
[paste function]
Write pytest unit tests covering:
- The happy path
- Empty input
- Edge case: [specific case you are worried about]
- Error conditions
Kimi K2 Instruct handles this pattern particularly well. It was trained with agentic coding workflows in mind, which means it produces self-consistent test suites rather than disconnected individual test functions that do not share fixtures or setup.
Refactoring is a different challenge than generation. The model needs to understand existing code before improving it, which makes context window size the decisive factor.
Paste the legacy module, describe the problem, and ask for a refactored version with the same external interface. Claude 4.5 Sonnet and GPT 4.1 both handle this well because they can hold the entire original function in attention while generating the replacement, rather than guessing at what was there before.
Bad prompts produce bad code. Good prompts produce code you can ship. The difference is almost entirely about how much context you give the model upfront.
A model generating code without context is guessing at your constraints. Telling it your language and framework halves the chance of it picking the wrong approach. Telling it the existing interface eliminates a whole class of integration bugs before they appear.
Minimal context prompt:
Write a function to parse CSV files
Effective context prompt:
Python 3.11, using the csv module (not pandas).
Write a function parse_csv(filepath: str) -> list[dict]
that reads a CSV with a header row and returns a list of dicts.
Handle FileNotFoundError and return an empty list if the file is empty.
The second prompt produces usable code. The first produces something plausible that may or may not fit your codebase.
💡 If the first response is wrong, do not just re-run the same prompt. Add one sentence explaining what was incorrect. Models respond far better to corrective follow-ups than repeated identical prompts.
One of the most significant practical gaps between Codex and today's models is not capability per token. It is how many tokens they can attend to at once.
Codex at 8K tokens could process roughly 400 to 600 lines of code. That works for individual functions but breaks down immediately for anything involving multiple files with shared types, API handlers referencing database models, or integration tests spanning several modules.
Granite 20B Code Instruct 8K matches Codex's context window but brings considerably more parameters and a more recent training corpus, making its per-token output substantially more accurate. Granite 8B Code Instruct 128K trades raw parameter count for reach: 128K tokens in a model built specifically for code tasks from the ground up.
For full-codebase operations, GPT 5 represents the current frontier, combining near-unlimited context with broad software engineering capability across every major language and framework.
The generation paradigm Codex established was: give context, predict tokens. The reasoning paradigm that followed adds a step before output: think through the problem first.
Models like DeepSeek R1 and GPT 5 work through a chain of intermediate thoughts before producing code. This makes a real difference for:
The reasoning trace itself is also valuable for code review. If the model explains that it chose a particular pattern to avoid a specific class of bug, you can verify that reasoning directly rather than auditing a black box output.
💡 Practical note: Reasoning models are slower and cost more per token. Use them for correctness-critical code. For boilerplate and tests, a fast model like Claude 4.5 Haiku is far more efficient.
Codex for code generation was a proof of concept that the entire industry validated and then raced past. The models available now do everything Codex did, with larger context windows, better reasoning, and more accurate outputs across a wider range of languages and frameworks.
If you have not used an AI code model seriously, the easiest place to start is a task you do every week but find tedious. Test generation, docstring writing, and boilerplate scaffolding all have a short feedback loop: you see immediately whether the output is useful. Start there, build intuition for what works, and push into harder tasks as you calibrate your prompting style.
Every model in the comparison table above is available right now on PicassoIA. No setup. No API keys. Pick one, paste a function, and see what it does in under a minute. Try GPT 5 for complex cross-file work, DeepSeek R1 when you need to see the reasoning, or Granite 8B Code Instruct 128K for a focused, fast code-only experience. The gap between "I have heard about AI coding tools" and "I use them every day" is a single afternoon of experimentation.
