#   #   ###   ####   ####  ### ###  #####  #   #  #####  #####
 ##  #  #   #  #     #       #   #   #      ##  #    #    #
 # # #  #####   ###  #       #   #   ####   # # #    #    ####
 #  ##  #   #      # #       #   #   #      #  ##    #    #
 #   #  #   #  ####   ####  ### ###  #####  #   #    #    #####

Fig. 1. "a lighthouse": an unedited generation from the final model, rendered through its own glyph atlas, showing lamp room, gallery, tapering shaft, rocky base. This prompt collapsed to scattered marks two training cycles earlier; the restoration is the direct result of the long-repetition schedule and rare-caption upsampling of §3. The reveal order here is presentational; the live demo replays the model's actual commit order.

1THE TASK

An ASCII picture is a point in V^(48×80) with |V| = 98: 96 printable ASCII characters plus [PAD] and [MASK]. Conditioned on a caption c, the model represents p(x|c) and samples from it. Two framing decisions drive everything downstream. First, characters are the tokens: generating pixels and converting afterward caps quality at the converter and can never exhibit character-level intent (choosing () for a curve because a human would). Second, the problem is closer to language modeling than to image synthesis, being discrete symbols at exact positions, which selects masked bidirectional modeling over both raster-order autoregression (3,840 sequential steps; a fictitious reading order over a 2D artifact) and continuous diffusion (a cell is / or \, never 0.6 of each).

2ARCHITECTURE

            input grid  48×80 tokens
                 |  embedding + 2D RoPE positions
                 v
   +---------- block × 8 -----------------------------+
   | x <- x + SelfAttention(LN(x))     8 heads          |
   | x <- x + CrossAttention(LN(x), caption tokens)     |
   | x <- x + FFN(LN(x))               512->2048->512   |
   +----------------------------------------------------+
                 |  LN -> Linear 512->98      (glyph head)
                 |  LN -> Linear 512->1       (critic head)
                 v
     per-cell glyph logits          per-cell "is this right?"

Fig. 2. 34.5M parameters: 8 pre-norm blocks, width 512. Dropout is 0.0, and that number is a measurement, not a default (§6, failure 7).

Positions. Factorized 2D rotary embeddings: half of each head's dimensions rotate by row index, half by column, so attention depends only on (Δrow, Δcol). Stroke knowledge is learned once, not per location, and the model runs on canvases up to 96×160 it never trained at, which is the entire super-resolution feature (anchor the source characters on a 2× canvas, mask the gaps, inpaint).

Conditioning. Prompts are embedded by a frozen CLIP text encoder; up to 24 token vectors feed cross-attention in every block (compositional addressability, so a hull cell can attend to "sailboat" while another region attends to "two"), and their pooled mean joins a global additive vector along with a mask-ratio embedding. The model always knows how far along the decode is, the analogue of a diffusion model being told its noise level. All conditioning projections are zero-initialized, so the pathway is an exact no-op at splice time: text conditioning was retrofitted onto a checkpoint trained without it, and later retrofits (the critic head) reuse the same trick.

  prompt "a lighthouse"                     mask ratio 0.62
          |                                       |
   frozen CLIP text encoder               sinusoidal embedding
          |                                       |
    24 token vectors ----> pooled mean ----> [ + ] ----> global vector
          |                                        (added to every cell)
          v
  cross-attention in every block
  (per-cell addressing: one cell attends
   to "lighthouse", another to "a")

Fig. 3. The conditioning pathway. Token vectors give compositional control through cross-attention; the pooled mean and the mask ratio steer globally, so the model always knows its own noise level.

Critic. A 513-parameter linear head predicts, per visible cell, "is this glyph correct?". It is trained for free from labels the self-context corruption already computes (§3), consumed at decode time to rank commitments and select revisions (Token-Critic [2]).

3TRAINING

The objective is masked restoration: hide a random fraction of cells, predict the originals given the survivors and the caption. Four modifications, each purchased with a documented failure:

• Space down-weighting, auto-calibrated. With 80 to 90% of cells being space, plain cross-entropy makes "space" the low-risk answer everywhere, which is the prior behind blank collapse. Space cells are weighted w = (1−f)/f for measured space fraction f (equal total gradient share for space and ink), recomputed per dataset because a constant tuned on one corpus is a measurement of that corpus in disguise.
• Glyph-aware soft labels + render-space loss. Targets are smoothed toward visually similar glyphs (predicting | for / costs less than @), and an auxiliary MSE compares the differentiably rendered expectation of the prediction with the rendered target, so the loss sees what the eye sees.
• Mask-ratio distribution biased to 1.0. Generation starts from a fully masked grid; training ratios are sine-warped toward high values with max exactly 1.0, or the first and hardest decode steps run out of distribution.
• Self-context training (SUNDAE-style). Half of all batches rebuild part of their visible context from the model's own sampled predictions, and the loss keeps scoring every originally-masked cell, including the self-committed ones. The sign of that last clause is load-bearing: an ablation that excluded committed cells from the loss taught the model that its own wrong commits are legitimate context to extend, and measurably worsened the artifact it was meant to fix (§6, failure 8).

  ground truth --mask 15-100%--> [ truth | ############ masked ]
                                              |
                            no-grad forward, sample every cell
                                              |
                       reveal a slice of the model's OWN samples
                                              |
              [ truth | model commits | ##### still masked ]
                                              |
                                      trained forward
                                              |
                loss on EVERY originally-masked cell,
                the self-committed ones included

Fig. 4. Self-context training. Inference always conditions on the model's own commits; this puts that state into the training distribution. Scoring the self-committed cells is the load-bearing clause (§6, failure 8): excluding them taught the model its own errors were legitimate context.

Curriculum. Three stages (procedural geometry, synthetic captioned subjects, human-made pieces) at decreasing learning rates, with earlier stages replayed into every later stage at non-trivial fractions (~20%): single-hop replay at 3% measurably erased geometry by the end of training. Rare captions are upsampled toward the ~1000-exposure threshold that knowledge-capacity scaling work [7] associates with full-fidelity storage, and the masked objective tolerates data repetition far beyond autoregressive norms, because random masking is itself per-epoch augmentation [6].

Fig. 5. Geometry-stage loss, six epochs, from the actual training log of the self-context run (space-weighted cross-entropy plus auxiliary terms). The train series is the in-epoch running mean, which resets at epoch boundaries; validation is evaluated on the EMA weights, which is why it can sit at or below the live-weight train curve. Validation improves monotonically except a single epoch-4 blip.

4THE DATA ENGINE

Captioned ASCII art does not exist at scale, so the training set is manufactured caption-first, so every label is correct by construction:

caption --> t2i --> trim --> flatten --> convert --> ink  --> CLIP --> (grid,
 bank      image    border   background  to glyphs   band    filter    caption)

Fig. 6. One generated image yields up to nine training samples: three style dialects (silhouette / outline / tonal, selected at inference by caption tags) × itself plus count and pair composites tiled from already-paid-for singles.

The engine is 20% generation and 80% quality control; its five documented failures (letterboxing that taught the model to draw frames, an ink-quota binarizer that blobified photographs, a 93%-chimera prompt bank, contrast enhancement that amplified paper texture into a canvas-wide film, dialects that collapsed into one on stroke-drawn subjects) are catalogued with the others in §6 and in the project's study guide. Tone has a resolution floor, since tone variety scales with cell count [9], so subjects whose tonal conversion inks under 8% of the grid route their untagged caption to the outline dialect instead: at 48×80, line structure is the legible strategy for sparse subjects.

5DECODING

MaskGIT-style iterative parallel unmasking: sample every masked cell, commit the most confident, repeat under a cosine schedule; classifier-free guidance ramps from 1 to full scale across scheduled steps. Two domain-specific pathologies required surgery, and both diagnoses came from probes rather than theory:

The blank attractor. On a sparse canvas the most confident early predictions are always spaces; committed blank becomes the context for more blank. Countered in training (space weighting, ratio max 1.0) and at decode (an annealed anti-space logit bias, exploration noise on commit order).

Parallel-commit edge echo. The cosine schedule commits the most cells in its final steps, and the cells left last are precisely the ambiguous ones: every column an unseen edge could fall on. Mass-committed in one parallel step, each samples a different answer, giving several offset copies of the same edge. An inpainting probe isolated the mechanism (visible half of a rectangle → clean pinned region, a band of competing edges exactly where the hidden edge could be), and the fix is a tail-only commit cap: the head of the decode runs free (layout forms under exploration and rising guidance), and once the mask ratio falls below 0.35 commits are capped at 8 cells per step, resolving the ambiguous tail sequentially. A/B at fixed seed: "a cross" went from 155-ink multi-stroke chaos to a perfect 28-ink cross; capping the whole decode instead collapsed sparse prompts to blank. The two regimes need opposite treatment, which is the finding.

  mask ratio  1.0 ------------------- 0.35 ------------------- 0.0
              |     HEAD: uncapped      |    TAIL: cap 8 / step  |
              |  exploration noise on   |  greedy + sequential;  |
              |  commit order; guidance |  each commit becomes   |
              |  rises 1 -> full scale; |  context for the next  |
              |  layout forms           |  decision              |
              +--------------------------------------------------+
               blank attractor lives      edge echo is born here:
               here: anti-space bias      the ambiguous cells are
               and noise keep early       committed last, and the
               ink alive                  cap serializes them

Fig. 7. The two-regime decode. Head and tail fail in opposite directions and need opposite treatment: freeing the head fixed blank collapse ("a dragon" 8 → 865 ink), capping the tail fixed the echo ("a cross" 155 → 28 ink), on the same checkpoint.

"a rectangle", uncapped (300 ink)

with tail cap (97 ink)

"a cross", uncapped (155 ink)

with tail cap (28 ink)

Fig. 8. The edge echo and its cure, real probe outputs: same checkpoint, same seed and guidance; the material change is the tail-only commit cap. Every offset stroke in the left panels is a separately-committed hypothesis for the same edge.

ground truth

model input (right half hidden)

completion

Fig. 9. The inpainting probe that separated "the model doesn't know shapes" from "the decode can't coordinate": shown the left half of a diamond, the capped decode completes it at 100% cell accuracy, 100% ink precision and recall. Knowledge was never the problem.

"a dragon" (tonal, best of 4 seeds)

"a diamond"

"a triangle"

Fig. 10. Final-model outputs, unedited. The dragon is the best of four seeds under the canonical decode, which is precisely the serve-time selection policy (§5): across those four seeds the same settings produced 90 to 979 ink cells, so single-seed judgments of a stochastic sampler mislead in both directions. The geometry pair is drawn after the full three-stage curriculum, so the primitives survived two later training stages intact (the replay result of §3).

Serve time. k candidates are drawn and re-ranked by CLIP over the rendered grids (best-of-k); the intermediate grids double as the site's materialize animation, so the visualization shows the algorithm itself running, unmodified.

6THE FAILURE CHRONICLE

Nine failures shaped the system. Each is stated as symptom → cause → fix; the lessons are the paper's real payload.

7FINDINGS, INCLUDING NEGATIVE ONES

Table 1. Every adopted setting was decided by fixed-seed A/B probes on standard prompts; two 3-0-literature-verified techniques failed those probes anyway. The probes, not the papers, are the authority.

Fig. 11. What a negative result looks like: "a dragon" under the Halton low-discrepancy commit order, at 2,635 ink cells using 94 of the 96 glyphs, pure speckle. Position-forced early commits sample from near-uniform marginals on a blank-dominated canvas, then texture grows around the noise. The same checkpoint drew coherent tonal art under confidence ordering.

8LIMITATIONS & FUTURE WORK

• Sparse-subject tonal generation was exposure-limited; the 30-epoch cycle restored the lighthouse (Fig. 1). Remaining tonal variance is seed-dominated (90 to 979 ink across four seeds for "a dragon" at identical settings, Fig. 10). The uncapped decode raises the worst-seed floor (575 vs 90) and best-of-k re-ranking selects the winners, so serving tonal prompts uncapped with k = 4 is the operative policy.
• Off-the-shelf CLIP scores ASCII structure at chance level [8], so the best-of-k re-ranker is weaker than it looks; an ASCII-aware encoder (or a CLIP fine-tuned on the ASCIIBench corpus) is the drop-in remedy.
• Preference tuning on served best-of-k picks (D3PO's closed form applies exactly to this corruption family [5]) and a trained multi-scale curriculum are the two parked bets with written un-park conditions.

9REFERENCES

1. [1] Chang et al., MaskGIT: Masked Generative Image Transformer, 2022.
2. [2] Lezama et al., Improved Masked Image Generation with Token-Critic, ECCV 2022. arXiv:2209.04439
3. [3] ReMDM: Remasking Discrete Diffusion Models, 2025. arXiv:2503.00307
4. [4] Besnier et al., Halton Scheduler for Masked Generative Image Transformer, 2025. arXiv:2503.17076
5. [5] D3PO: Preference-Based Alignment of Discrete Diffusion Models, 2025. arXiv:2503.08295
6. [6] Muennighoff et al., Scaling Data-Constrained Language Models, 2023. arXiv:2305.16264
7. [7] Allen-Zhu & Li, Physics of Language Models 3.3: Knowledge Capacity Scaling Laws, 2024. arXiv:2404.05405
8. [8] Luo et al., ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text, 2025. arXiv:2512.04125
9. [9] Xu, Zhang & Wong, Structure-based ASCII Art, SIGGRAPH Asia 2010.