Add UL2 data sampling and pretraining - #358

Open

janEbert wants to merge 122 commits into

bigscience-workshop:mainfrom

Commits on Dec 13, 2022

Fix `PretrainedFromHF` tokenizer with T5 training
janEbert
committed
b2fc665
Allow passing existing casual attention masks
janEbert
committed
13becf1
Refactor masked LM sampling style selection
janEbert
committed
7f50532
Add more masked LM sampling styles
janEbert
committed
d8db189
Allow Prefix-LM style masked LM
janEbert
committed
006c4e9
Add UL2 pretraining for T5 model
janEbert
committed
f802317
Refactor span merging
janEbert
committed
deed87f
Support UL2 for decoder-only models
janEbert
committed
728e076
Unconditionally use safe maximum sequence length
janEbert
committed
42ece6b

Commits on Dec 14, 2022

Add custom exceptions
janEbert
committed
d18f84e
Error out on too long sequences
janEbert
committed
fa5aa68
Remove additional sequence truncation
janEbert
committed
c7d8a8b
Prefer array-from-list creation
janEbert
committed
c722516

Commits on Jan 2, 2023

Remove redundant imports
janEbert
committed
69f6e70

Commits on Jan 3, 2023

Fix not inserting prefixes
janEbert
committed
f08a104
Do not insert `extra_id` tokens for PrefixLM task
janEbert
committed
d2fd03e
Document `max_seq_length_dec` argument
janEbert
committed
daf52cc
Skip redundant computations
janEbert
committed
04be590
Fix PrefixLM mean location
janEbert
committed
7bc5a87
Pad decoder-only inputs to same length
janEbert
committed
775e99d
Fix decoder-only attention mask shape
janEbert
committed
538c30b

Commits on Jan 23, 2023

Document index set selection for PrefixLM masking
janEbert
committed
ba4476c
Fix `max_ngrams` for normal sampling style
janEbert
committed
678fbdc
Do not limit `max_predictions_per_seq`
janEbert
committed
00479e5
Calculate and use amount of filtered tokens
janEbert
committed
795caef
Document normal sampling style
janEbert
committed
689e15f
Fix PrefixLM possible spans calculation
janEbert
committed
e44d0e4

Commits on Jan 24, 2023

Use binary search for PrefixLM first tail index
janEbert
committed
075f05f
Calculate n-gram indices lazily
janEbert
committed
6bc7471
Fix code style
janEbert
committed
a105f32
Prefer list comprehensions
janEbert
committed
f0fe282

Commits on Feb 14, 2023

Allow recognizing when UL2 is used
janEbert
committed
11bd6db
Support UL2 tokens for all tokenizers
janEbert
committed
43eee93
Support `<extra_id>` tokens for GPT tokenizer
janEbert
committed
6686f04
Fix tokenizer vocab access
janEbert
committed
f6128c6
Revert inheriting from `T5Dataset`
janEbert
committed
8f48763
Fix GPT tokenizer special token handling
janEbert
committed
7f99a12
Do inherit from `torch.utils.data.Dataset`
janEbert
committed
535a306
Add whitespace
janEbert
committed
db623b3
Allow selectively disabling denoiser token
janEbert
committed
ef72280
Allow not replacing masks with sentinel tokens
janEbert
committed
001b50c
Support not adding mask tokens in span corruption
janEbert
committed
23c052f

Commits on Feb 15, 2023

Fix expected number of added tokens
janEbert
committed
0f4fd3f

Commits on Feb 16, 2023

Fix non-masked data
janEbert
committed
da1f4e9
Fix unclear wording
janEbert
committed
55320ea

Commits on Feb 17, 2023

Adjust code style
janEbert
committed
5d27b27
Fix covered index skipping
janEbert
committed
23181ab
Prepend objective token before truncating
janEbert
committed
6032cc6
Automatically truncate sequences for decoder-only
janEbert
committed
c9c336f
Fix covered span skipping fix
janEbert
committed
b8003cb
Make `build_index_mappings` public
janEbert
committed
e3d91a6
Refactor getting sample
janEbert
committed
e61e78f
Add sample packing to T5 dataset
janEbert
committed
c3b0a55
Add sample packing to UL2 dataset
janEbert
committed
c4d748b
Fix typo and comment placement
janEbert
committed
689b57e
Fix not supplying `--pack-samples` argument
janEbert
committed
af204e7
Add support for UL2R-style implementation
janEbert
committed
78eb035
Fix T5 dataset packing
janEbert
committed
c03eed4

Commits on Feb 22, 2023

Refactor `get_sample` to return a list
janEbert
committed
9e84f06
Fix T5 sample packing
janEbert
committed
5e2b4f5
Fix UL2 sample packing
janEbert
committed
e2a0c36
Refactor samples dict creation
janEbert
committed
c2884c8

Commits on Feb 23, 2023

Fix desired seq length
janEbert
committed
7eb7923
Fix padding removal
janEbert
committed
dd4c0d0
Allow repeating UL2 prompt token when packing
janEbert
committed
58148f8
Allow packing different denoisers together
janEbert
committed
c41fecd
Refactor sample packing functions
janEbert
committed
057bb47
Repeat prompt by default when packing UL2
janEbert
committed
e2062b7
Support pipelining for decoder-only model
janEbert
committed
d31b89f

Commits on Feb 24, 2023

Fix GPT tokenizer vocab size query
janEbert
committed
17dca4f
Handle possibly empty list
janEbert
committed
bf9b1eb

Commits on Feb 27, 2023

Fix no newline at EOF
janEbert
committed
c4aa4cd
Allow full prefix Prefix-LM attention sampling
janEbert
committed
8d7a0df
Support PrefixLM models
janEbert
committed
9bd6e1e
Allow setting number of few-shot examples
janEbert
committed
ba4ab49
Update task/dataset name
janEbert
committed
9f53171

Commits on Feb 28, 2023

Do not remove last token
janEbert
committed
5b63d0b
Fix PrefixLM contexts
janEbert
committed
639b71d
Fix module refactor
janEbert
committed
127d1e4
Fix possible `TypeError`
janEbert
committed
1bb788d
Optionally add prefix tokens
janEbert
committed
cf5965a
Automatically add UL2 tokens
janEbert
committed
a538238

Commits on Mar 1, 2023

Fix context lengths batch chunking
janEbert
committed
3a8bc35
Allow different models to be loaded
janEbert
committed
6f0e33a

Commits on Mar 2, 2023

Fix context batch size padding
janEbert
committed
9c4c718

Commits on Mar 7, 2023

Add xPos embeddings
janEbert
committed
754cf21
Add optional UL2 normal distribution scaling
janEbert
committed
08b0eaf
Allow evaluating encoder-decoder models
janEbert
committed
15622d2

Commits on Mar 9, 2023

Fix not passing `scale_normal_std`
janEbert
committed
e5a6169
Add T5-style GLU layers
janEbert
committed
d583fe9
Rename xPos embedding class
janEbert
committed
ad7de7e
Integrate xPos embedding
janEbert
committed
81a68f7
Handle xPos embedding
janEbert
committed
46e145d
Do not use bias for 2nd MLP layer if using T5 GLU
janEbert
committed
482f0ea
Fix T5 GLU constructor arguments
janEbert
committed
4385f7b
Refactor samples dict creation
janEbert
committed
2d24b13
Move callees under caller
janEbert
committed
bd461f5

Commits on Mar 10, 2023

Handle empty context
janEbert
committed
35b2956
Handle more possible model types
janEbert
committed
f0171e0
Fix fully truncated contexts with prefix tokens
janEbert
committed
92158d8
Make T5 GLU checks safer
janEbert
committed
3b7692f

Commits on Mar 20, 2023

Improve import code style
janEbert
committed
b37d3ee
Refactor dummy barriers
janEbert
committed
5959e89
Refactor file name creation
janEbert
committed
ce8c1a5
Allow packing only full documents
janEbert
committed
3e52966
Use full-doc packing for T5-style datasets
janEbert
committed
23efa88
Fix trying to all-reduce non-existent bias
janEbert
committed
88eb98a

Commits on Mar 21, 2023

Fix truncating packed sequences without padding
janEbert
committed
59e8451

Commits on Mar 24, 2023

Speed up packed dataset indexing
janEbert
committed
24d46ff

Commits on Apr 3, 2023

Try to exit padding removal early
janEbert
committed
600542d

Commits on Apr 4, 2023

Fix xPos embedding
janEbert
committed
58831d2

Commits on Apr 13, 2023

Fix padding loss mask
janEbert
committed
fe45cea
Handle failure mode regarding non-DS checkpoints
janEbert
committed
15e7b98

Commits on Jun 7, 2023

Fix decoder-only and no-mask-tokens seq lengths
janEbert
committed
ae45a9e
Omit second objective token if without mask tokens
janEbert
committed
0c91b96
Fix NumPy deprecations
janEbert
committed
0c246c4

Commits on Jun 26, 2023

Fix supplied arguments
janEbert
committed
7ce8635
Do not add separator if S-denoising
janEbert
committed
7290181
Fix caching error
janEbert
committed
628d847

Commits on Jun 29, 2023

Fix number of labels calculation for decoder-only
janEbert
committed
9c727e7
Do not automatically add <EOS> token when packing
janEbert
committed
4ffa951
Allow silently ignoring causal attention mask
janEbert
committed
ff5787e