Research question
"What are BERT's two pre-training objectives, and why does bidirectional context matter?"
A useful answer separates the paper's two objectives from the reason its context setup matters.
Masked language modeling
BERT predicts randomly masked input tokens from their surrounding context. This lets the representation combine information from both the left and right sides of a token.
BERT paper, PDF pp. 1-2Next sentence prediction
The second task trains BERT to decide whether one sentence follows another, pretraining relationships that matter for sentence-pair tasks.
BERT paper, PDF p. 4Deep bidirectional context
BERT conditions on left and right context across every layer. The authors contrast this with unidirectional models, whose one-sided context can limit token-level tasks such as question answering.
BERT paper, PDF p. 1
