diff options
author | Mark H Weaver <mhw@netris.org> | 2013-04-03 04:22:04 -0400 |
---|---|---|
committer | Mark H Weaver <mhw@netris.org> | 2013-04-04 21:40:28 -0400 |
commit | cdd3d6c9f423d5b95f05193fe3c27d50b56957e9 (patch) | |
tree | 66e9f974fea9b9c4c1aa1d813dcf489109f474f7 /doc/ref/api-io.texi | |
parent | 45c0878b8665182f06a917e391169031c1dc7db6 (diff) | |
download | guile-cdd3d6c9f423d5b95f05193fe3c27d50b56957e9.tar.gz |
Improve handling of Unicode byte-order marks (BOMs).
* libguile/ports-internal.h (struct scm_port_internal): Add new members
'at_stream_start_for_bom_read' and 'at_stream_start_for_bom_write'.
(SCM_UNICODE_BOM): New macro.
(scm_i_port_iconv_descriptors): Add 'mode' parameter to prototype.
* libguile/ports.c (scm_new_port_table_entry): Initialize
'at_stream_start_for_bom_read' and 'at_stream_start_for_bom_write'.
(get_iconv_codepoint): Pass new 'mode' parameter to
'scm_i_port_iconv_descriptors'.
(get_codepoint): After reading a codepoint at stream start, record
that we're no longer at stream start, and consume a BOM where
appropriate.
(scm_seek): Set the stream start flags according to the new position.
(looking_at_bytes): New static function.
(scm_utf8_bom, scm_utf16be_bom, scm_utf16le_bom, scm_utf32be_bom,
scm_utf32le_bom): New static const arrays.
(decide_utf16_encoding, decide_utf32_encoding): New static functions.
(scm_i_port_iconv_descriptors): Add new 'mode' parameter. If the
specified encoding is UTF-16 or UTF-32, make that precise by deciding
what byte order to use, and construct iconv descriptors based on the
precise encoding.
(scm_i_set_port_encoding_x): Record that we are now at stream start.
Do not open the new iconv descriptors immediately; let them be
initialized lazily.
* libguile/print.c (display_string_using_iconv): Record that we're no
longer at stream start. Write a BOM if appropriate.
* doc/ref/api-io.texi (BOM Handling): New node.
* test-suite/tests/ports.test ("set-port-encoding!, wrong encoding"):
Adapt test to cope with the fact that 'set-port-encoding!' does not
immediately open the iconv descriptors.
(bv-read-test): New procedure.
("unicode byte-order marks (BOMs)"): New test prefix.
Diffstat (limited to 'doc/ref/api-io.texi')
-rw-r--r-- | doc/ref/api-io.texi | 81 |
1 files changed, 80 insertions, 1 deletions
diff --git a/doc/ref/api-io.texi b/doc/ref/api-io.texi index 8c974be5f..9c3e1fc27 100644 --- a/doc/ref/api-io.texi +++ b/doc/ref/api-io.texi @@ -1,7 +1,7 @@ @c -*-texinfo-*- @c This is part of the GNU Guile Reference Manual. @c Copyright (C) 1996, 1997, 2000, 2001, 2002, 2003, 2004, 2007, 2009, -@c 2010, 2011 Free Software Foundation, Inc. +@c 2010, 2011, 2013 Free Software Foundation, Inc. @c See the file guile.texi for copying conditions. @node Input and Output @@ -19,6 +19,7 @@ * Port Types:: Types of port and how to make them. * R6RS I/O Ports:: The R6RS port API. * I/O Extensions:: Using and extending ports in C. +* BOM Handling:: Handling of Unicode byte order marks. @end menu @@ -2373,6 +2374,84 @@ Set using @end table +@node BOM Handling +@subsection Handling of Unicode byte order marks. +@cindex BOM +@cindex byte order mark + +This section documents the finer points of Guile's handling of Unicode +byte order marks (BOMs). A byte order mark (U+FEFF) is typically found +at the start of a UTF-16 or UTF-32 stream, to allow readers to reliably +determine the byte order. Occasionally, a BOM is found at the start of +a UTF-8 stream, but this is much less common and not generally +recommended. + +Guile attempts to handle BOMs automatically, and in accordance with the +recommendations of the Unicode Standard, when the port encoding is set +to @code{UTF-8}, @code{UTF-16}, or @code{UTF-32}. In brief, Guile +automatically writes a BOM at the start of a UTF-16 or UTF-32 stream, +and automatically consumes one from the start of a UTF-8, UTF-16, or +UTF-32 stream. + +As specified in the Unicode Standard, a BOM is only handled specially at +the start of a stream, and only if the port encoding is set to +@code{UTF-8}, @code{UTF-16} or @code{UTF-32}. If the port encoding is +set to @code{UTF-16BE}, @code{UTF-16LE}, @code{UTF-32BE}, or +@code{UTF-32LE}, then BOMs are @emph{not} handled specially, and none of +the special handling described in this section applies. + +@itemize @bullet +@item +To ensure that Guile will properly detect the byte order of a UTF-16 or +UTF-32 stream, you must perform a textual read before any writes, seeks, +or binary I/O. Guile will not attempt to read a BOM unless a read is +explicitly requested at the start of the stream. + +@item +If a textual write is performed before the first read, then an arbitrary +byte order will be chosen. Currently, big endian is the default on all +platforms, but that may change in the future. If you wish to explicitly +control the byte order of an output stream, set the port encoding to +@code{UTF-16BE}, @code{UTF-16LE}, @code{UTF-32BE}, or @code{UTF-32LE}, +and explicitly write a BOM (@code{#\xFEFF}) if desired. + +@item +If @code{set-port-encoding!} is called in the middle of a stream, Guile +treats this as a new logical ``start of stream'' for purposes of BOM +handling, and will forget about any BOMs that had previously been seen. +Therefore, it may choose a different byte order than had been used +previously. This is intended to support multiple logical text streams +embedded within a larger binary stream. + +@item +Binary I/O operations are not guaranteed to update Guile's notion of +whether the port is at the ``start of the stream'', nor are they +guaranteed to produce or consume BOMs. + +@item +For ports that support seeking (e.g. normal files), the input and output +streams are considered linked: if the user reads first, then a BOM will +be consumed (if appropriate), but later writes will @emph{not} produce a +BOM. Similarly, if the user writes first, then later reads will +@emph{not} consume a BOM. + +@item +For ports that do not support seeking (e.g. pipes, sockets, and +terminals), the input and output streams are considered +@emph{independent} for purposes of BOM handling: the first read will +consume a BOM (if appropriate), and the first write will @emph{also} +produce a BOM (if appropriate). However, the input and output streams +will always use the same byte order. + +@item +Seeks to the beginning of a file will set the ``start of stream'' flags. +Therefore, a subsequent textual read or write will consume or produce a +BOM. However, unlike @code{set-port-encoding!}, if a byte order had +already been chosen for the port, it will remain in effect after a seek, +and cannot be changed by the presence of a BOM. Seeks anywhere other +than the beginning of a file clear the ``start of stream'' flags. +@end itemize + @c Local Variables: @c TeX-master: "guile.texi" @c End: |