Repository navigation
Character and string token definitions need updating. #626
Description
Activity
See also rust-lang/rust#62865
rust-lang/rust#118699 (comment)
should be helpful.The current description says that forms like'a'bare acceptable as aBYTE_LITERALwith a suffix, but in fact they're rejected (to avoid confusion with twoLIFETIME_LABELtokens).The current description says that forms like
'ab'care acceptable as twoLIFETIME_LABELtokens, but in fact they're rejected ("character literal may only contain one codepoint"; thecis taken as a suffix).Perhaps this could be documented via another reserved form.
A form like
b"\u{00a0}"is rejected at lexing time ("unicode escape in byte string").But as it doesn't match either
BYTE_STRING_LITERALorRESERVED_TOKEN_DOUBLE_QUOTE, the current description says there's a valid tokenisation as the identifierbfollowed by"\u{00a0}".So if we keep on with the current mechanism for documenting such rejected tokens, I think we'd need yet more reserved forms.
There are probably other similar cases. I think after rust-lang/rust#119172 a
C string literal containing aNULis one.
There are multiple issues here. Some of this has changed in 1.37 via rust-lang/rust#60793.
RAW_BYTE_STRING_LITERALno longer allows bare CR (new 1.37). Input format #1459"Raw string" and "raw byte string" needs to be updated that CRLF is converted to LF (new 1.37). Input format #1459
Several tokens need to sync the English text with the "Lexer" definition.
STRING_LITERALindicates several rules (like isolated CR's are not allowed), but the text does not mention any of those restrictions.CHAR_LITERALsays "single Unicode character…except U+0027" which is not complete.RAW_STRING_LITERALdoes not allow bare CR's.BYTE_LITERALescapes are not described.BYTE_STRING_LITERALrestrictions are not described.Typo in
RAW_BYTE_STRING_CONTENT, points toRAW_STRING_CONTENTwhen it should beRAW_BYTE_STRING_CONTENT. Fixes minor errors #818I cannot find anywhere that mentions CRLF in a string is converted to LF. Am I blind? Input format #1459
The description for string continuations says "
\immediately beforeU+000A", but it can also be before CRLF. How should this be handled? I haven't looked at how it is implemented, but are all CRLF's translated everywhere? Should there just be a blanket statement somewhere about this, to avoid having to discuss it in every string literal definition? Input format #1459I may be missing some things here. Need to very thoroughly review everything to make sure it is correct and up-to-date with the changes from 60793.