validate roberta.base with wikitext-103

$ python3 fairseq_cli/validate.py examples/language_model/data-bin/wikitext-103 --path ~/Downloads/roberta.base/model.pt --task masked_lm --max-tokens 512

2022-08-02 14:16:30 | INFO | fairseq.data.data_utils | loaded 3,760 examples from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 14:16:30 | INFO | fairseq.tasks.masked_lm | loaded 580 blocks from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 14:43:30 | INFO | valid | valid on 'valid' subset | loss 2.055 | ppl 4.16 | wps 0 | wpb 246983 | bsz 580

$ python3 fairseq_cli/validate.py examples/language_model/data-bin/wikitext-103 --path ~/Downloads/roberta.base/model.pt --task masked_lm --max-tokens 1024

2022-08-02 15:04:04 | INFO | fairseq.data.data_utils | loaded 3,760 examples from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 15:04:04 | INFO | fairseq.tasks.masked_lm | loaded 580 blocks from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 15:31:16 | INFO | valid | valid on 'valid' subset | loss 2.055 | ppl 4.16 | wps 0 | wpb 246983 | bsz 580

$ python3 fairseq_cli/validate.py examples/language_model/data-bin/wikitext-103 --path ~/Downloads/roberta.base/model.pt --task masked_lm --max-tokens 256

2022-08-02 15:33:29 | INFO | fairseq.data.data_utils | loaded 3,760 examples from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 15:33:29 | INFO | fairseq.tasks.masked_lm | loaded 580 blocks from: examples/language_model/data-bin/wikitext-103/valid

Traceback (most recent call last):

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq_cli/validate.py", line 153, in <module>

    cli_main()

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq_cli/validate.py", line 147, in cli_main

    distributed_utils.call_main(

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq/distributed/utils.py", line 369, in call_main

    main(cfg, **kwargs)

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq_cli/validate.py", line 93, in main

    itr = task.get_batch_iterator(

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq/tasks/fairseq_task.py", line 295, in get_batch_iterator

    batch_sampler = dataset.batch_by_size(

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq/data/base_wrapper_dataset.py", line 61, in batch_by_size

    return self.dataset.batch_by_size(

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq/data/fairseq_dataset.py", line 145, in batch_by_size

    return data_utils.batch_by_size(

  File "/Users/jason_chou/Documents/GitHub/fairseq/fairseq/data/data_utils.py", line 340, in batch_by_size

    return batch_by_size_fn(

  File "fairseq/data/data_utils_fast.pyx", line 108, in fairseq.data.data_utils_fast.batch_by_size_fn

    cpdef list batch_by_size_fn(

  File "fairseq/data/data_utils_fast.pyx", line 123, in fairseq.data.data_utils_fast.batch_by_size_fn

    return batch_by_size_vec(indices, num_tokens_vec, max_tokens,

  File "fairseq/data/data_utils_fast.pyx", line 30, in fairseq.data.data_utils_fast.batch_by_size_vec

    assert max_tokens <= 0 or np.max(num_tokens_vec) <= max_tokens, (

AssertionError: Sentences lengths should not exceed max_tokens=256

The first two runs are probably equivalent. I need to figure out how to actually change the context length. Exact same with max_tokens=2048 and 4096 as well:

$ python3 fairseq_cli/validate.py examples/language_model/data-bin/wikitext-103 --path ~/Downloads/roberta.base/model.pt --task masked_lm --max-tokens 2048

2022-08-02 15:48:54 | INFO | fairseq.data.data_utils | loaded 3,760 examples from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 15:48:55 | INFO | fairseq.tasks.masked_lm | loaded 580 blocks from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 16:16:02 | INFO | valid | valid on 'valid' subset | loss 2.055 | ppl 4.16 | wps 0 | wpb 246983 | bsz 580  

$ python3 fairseq_cli/validate.py examples/language_model/data-bin/wikitext-103 --path ~/Downloads/roberta.base/model.pt --task masked_lm --max-tokens 4096

2022-08-02 16:18:01 | INFO | fairseq.data.data_utils | loaded 3,760 examples from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 16:18:01 | INFO | fairseq.tasks.masked_lm | loaded 580 blocks from: examples/language_model/data-bin/wikitext-103/valid

2022-08-02 16:45:08 | INFO | valid | valid on 'valid' subset | loss 2.055 | ppl 4.16 | wps 0 | wpb 246983 | bsz 580